Attachment 664728 Details for Bug 887751 – engine.log

engine.log

engine.mtu_30.log (text/x-log), 4.30 MB, created by Meni Yakove on 2012-12-17 08:58:38 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Meni Yakove

Created: 2012-12-17 08:58:38 UTC

Size: 4.30 MB

patch

obsolete

>2012-12-17 08:16:28,502 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [7b78f9c5] FINISH, ResetIrsVDSCommand, log id: 5bc76910
>2012-12-17 08:16:28,514 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [7b78f9c5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 34c09940
>2012-12-17 08:16:28,515 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [7b78f9c5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:16:28,515 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [7b78f9c5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 34c09940
>2012-12-17 08:16:28,587 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:16:28,598 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4489
>2012-12-17 08:16:28,599 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4489, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4225de10
>2012-12-17 08:16:28,622 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] spmStart polling started: taskId = 52fec17c-28ea-4404-b5d6-547238e0502c
>2012-12-17 08:16:29,644 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] spmStart polling ended: taskId = 52fec17c-28ea-4404-b5d6-547238e0502c task status = finished
>2012-12-17 08:16:29,662 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] spmStart polling ended, spm status: SPM
>2012-12-17 08:16:29,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=52fec17c-28ea-4404-b5d6-547238e0502c), log id: 591a05be
>2012-12-17 08:16:29,682 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] FINISH, HSMClearTaskVDSCommand, log id: 591a05be
>2012-12-17 08:16:29,682 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3b6f2aa5, log id: 4225de10
>2012-12-17 08:16:29,695 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:16:29,710 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:16:29,711 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4f9e16d3
>2012-12-17 08:16:29,732 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4f9e16d3
>2012-12-17 08:16:29,732 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2feb4141
>2012-12-17 08:16:29,732 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [3cf5d2fd] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:16:29,732 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 13ec9f7b
>2012-12-17 08:16:29,732 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:16:29,734 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1073bebc
>2012-12-17 08:16:29,753 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1073bebc
>2012-12-17 08:16:29,753 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 13ec9f7b
>2012-12-17 08:16:29,753 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [3cf5d2fd] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:16:34,177 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:16:38,660 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [5ee35c25] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:16:38,799 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [f4188d3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:16:38,838 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [f4188d3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4358f48e
>2012-12-17 08:16:38,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [f4188d3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2fd30bb7
>2012-12-17 08:16:38,859 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [f4188d3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:16:39,024 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [f4188d3] FINISH, SpmStopVDSCommand, log id: 2fd30bb7
>2012-12-17 08:16:39,039 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [f4188d3] FINISH, ResetIrsVDSCommand, log id: 4358f48e
>2012-12-17 08:16:39,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [f4188d3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: dd5fbae
>2012-12-17 08:16:39,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [f4188d3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:16:39,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [f4188d3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: dd5fbae
>2012-12-17 08:16:44,346 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [58c4b5b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:16:49,096 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [740958cc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:16:49,110 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [740958cc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4490
>2012-12-17 08:16:49,112 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [740958cc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4490, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1fb80809
>2012-12-17 08:16:49,134 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [740958cc] spmStart polling started: taskId = b1a15a37-79f1-4ac5-b237-8e13829268ee
>2012-12-17 08:16:50,157 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [740958cc] spmStart polling ended: taskId = b1a15a37-79f1-4ac5-b237-8e13829268ee task status = finished
>2012-12-17 08:16:50,175 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [740958cc] spmStart polling ended, spm status: SPM
>2012-12-17 08:16:50,177 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [740958cc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b1a15a37-79f1-4ac5-b237-8e13829268ee), log id: 49e9614e
>2012-12-17 08:16:50,196 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [740958cc] FINISH, HSMClearTaskVDSCommand, log id: 49e9614e
>2012-12-17 08:16:50,196 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [740958cc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78f36dca, log id: 1fb80809
>2012-12-17 08:16:50,470 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [740958cc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:16:51,001 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [676a97fd] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:16:51,002 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 66f28248
>2012-12-17 08:16:51,053 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [740958cc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:16:51,383 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-43) [42c50c6a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:16:51,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [42c50c6a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1d18de24
>2012-12-17 08:16:51,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [42c50c6a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 13cf339e
>2012-12-17 08:16:51,450 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [42c50c6a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:16:51,588 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [42c50c6a] FINISH, SpmStopVDSCommand, log id: 13cf339e
>2012-12-17 08:16:51,603 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [42c50c6a] FINISH, ResetIrsVDSCommand, log id: 1d18de24
>2012-12-17 08:16:51,614 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [42c50c6a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4b33bc
>2012-12-17 08:16:51,614 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [42c50c6a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:16:51,614 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [42c50c6a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4b33bc
>2012-12-17 08:16:51,758 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [676a97fd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:16:51,771 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [676a97fd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4491
>2012-12-17 08:16:51,773 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4491, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 56ae90dd
>2012-12-17 08:16:51,796 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] spmStart polling started: taskId = 0abaa22a-a784-4345-b3fe-41e6859318ac
>2012-12-17 08:16:52,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] spmStart polling ended: taskId = 0abaa22a-a784-4345-b3fe-41e6859318ac task status = finished
>2012-12-17 08:16:52,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] spmStart polling ended, spm status: SPM
>2012-12-17 08:16:52,842 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0abaa22a-a784-4345-b3fe-41e6859318ac), log id: 76a203d9
>2012-12-17 08:16:52,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, HSMClearTaskVDSCommand, log id: 76a203d9
>2012-12-17 08:16:52,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6aad868, log id: 56ae90dd
>2012-12-17 08:16:52,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [676a97fd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:16:52,903 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:16:52,905 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 701085c2
>2012-12-17 08:16:52,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 701085c2
>2012-12-17 08:16:52,924 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 66f28248
>2012-12-17 08:16:52,924 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [676a97fd] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:16:52,924 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3cc57f94
>2012-12-17 08:16:52,924 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:16:52,926 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 63b628a9
>2012-12-17 08:16:52,944 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 63b628a9
>2012-12-17 08:16:52,944 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3cc57f94
>2012-12-17 08:16:52,944 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [676a97fd] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:16:54,612 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [284eeca1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:17:01,810 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [412bf4c7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:17:01,988 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [16233e3c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:17:02,094 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 42717f3
>2012-12-17 08:17:02,095 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2d8a6a03
>2012-12-17 08:17:02,115 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:17:02,272 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] FINISH, SpmStopVDSCommand, log id: 2d8a6a03
>2012-12-17 08:17:02,288 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] FINISH, ResetIrsVDSCommand, log id: 42717f3
>2012-12-17 08:17:02,298 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7b97a385
>2012-12-17 08:17:02,298 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [16233e3c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:17:02,298 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7b97a385
>2012-12-17 08:17:04,839 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6995ddd2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:17:12,359 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [28cae87e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:17:12,447 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [28cae87e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4492
>2012-12-17 08:17:12,448 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [28cae87e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4492, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 15994090
>2012-12-17 08:17:12,470 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [28cae87e] spmStart polling started: taskId = 04d01d20-76bd-4401-a5d9-d6ff76d6d6e2
>2012-12-17 08:17:13,492 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [28cae87e] spmStart polling ended: taskId = 04d01d20-76bd-4401-a5d9-d6ff76d6d6e2 task status = finished
>2012-12-17 08:17:13,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [28cae87e] spmStart polling ended, spm status: SPM
>2012-12-17 08:17:13,511 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [28cae87e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=04d01d20-76bd-4401-a5d9-d6ff76d6d6e2), log id: 1ffcfe67
>2012-12-17 08:17:13,529 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [28cae87e] FINISH, HSMClearTaskVDSCommand, log id: 1ffcfe67
>2012-12-17 08:17:13,529 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [28cae87e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1aa22c99, log id: 15994090
>2012-12-17 08:17:13,554 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [28cae87e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:17:13,609 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [71ec072c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:17:13,609 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 390ef54a
>2012-12-17 08:17:13,615 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [28cae87e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:17:13,864 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [48110f6e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:17:13,997 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [48110f6e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 24c814a4
>2012-12-17 08:17:13,999 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [48110f6e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 11c984e3
>2012-12-17 08:17:14,018 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [48110f6e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:17:14,241 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [48110f6e] FINISH, SpmStopVDSCommand, log id: 11c984e3
>2012-12-17 08:17:14,279 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [48110f6e] FINISH, ResetIrsVDSCommand, log id: 24c814a4
>2012-12-17 08:17:14,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [48110f6e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44eeeac
>2012-12-17 08:17:14,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [48110f6e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:17:14,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [48110f6e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44eeeac
>2012-12-17 08:17:14,345 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:17:14,355 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4493
>2012-12-17 08:17:14,356 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4493, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 46c64d52
>2012-12-17 08:17:14,378 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] spmStart polling started: taskId = ba137dd1-b926-4f3f-a17a-a9a55be53ec6
>2012-12-17 08:17:15,398 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] spmStart polling ended: taskId = ba137dd1-b926-4f3f-a17a-a9a55be53ec6 task status = finished
>2012-12-17 08:17:15,417 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] spmStart polling ended, spm status: SPM
>2012-12-17 08:17:15,418 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ba137dd1-b926-4f3f-a17a-a9a55be53ec6), log id: 31277d4a
>2012-12-17 08:17:15,436 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, HSMClearTaskVDSCommand, log id: 31277d4a
>2012-12-17 08:17:15,436 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@17929e7c, log id: 46c64d52
>2012-12-17 08:17:15,620 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:17:16,098 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:17:16,103 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2abbb816
>2012-12-17 08:17:16,125 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2abbb816
>2012-12-17 08:17:16,125 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 390ef54a
>2012-12-17 08:17:16,125 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [71ec072c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:17:16,125 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: a32c245
>2012-12-17 08:17:16,125 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:17:16,127 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4cf02e36
>2012-12-17 08:17:16,145 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4cf02e36
>2012-12-17 08:17:16,145 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a32c245
>2012-12-17 08:17:16,145 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [71ec072c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:17:24,396 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [3637a35] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:17:24,571 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-68) [406132e6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:17:24,625 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5b2310c2
>2012-12-17 08:17:24,627 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 402ad8a5
>2012-12-17 08:17:24,647 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [406132e6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:17:25,204 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, SpmStopVDSCommand, log id: 402ad8a5
>2012-12-17 08:17:25,646 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, ResetIrsVDSCommand, log id: 5b2310c2
>2012-12-17 08:17:25,657 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 48c978e0
>2012-12-17 08:17:25,657 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:17:25,657 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 48c978e0
>2012-12-17 08:17:25,859 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [508b37e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:17:35,893 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [380e9d7e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:17:35,952 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [380e9d7e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4494
>2012-12-17 08:17:35,953 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [380e9d7e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4494, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: fd4449e
>2012-12-17 08:17:35,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [380e9d7e] spmStart polling started: taskId = 94e60bdc-baad-477e-8735-7d19a04d9e32
>2012-12-17 08:17:37,000 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [380e9d7e] spmStart polling ended: taskId = 94e60bdc-baad-477e-8735-7d19a04d9e32 task status = finished
>2012-12-17 08:17:37,018 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [380e9d7e] spmStart polling ended, spm status: SPM
>2012-12-17 08:17:37,020 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [380e9d7e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=94e60bdc-baad-477e-8735-7d19a04d9e32), log id: 20b8102f
>2012-12-17 08:17:37,040 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [380e9d7e] FINISH, HSMClearTaskVDSCommand, log id: 20b8102f
>2012-12-17 08:17:37,040 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [380e9d7e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@142e8e71, log id: fd4449e
>2012-12-17 08:17:37,048 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [380e9d7e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:17:37,077 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [5a4d3cd3] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:17:37,077 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 357be1d1
>2012-12-17 08:17:37,114 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [380e9d7e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:17:37,203 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:17:37,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6e09901
>2012-12-17 08:17:37,273 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5549e288
>2012-12-17 08:17:37,292 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:17:37,439 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SpmStopVDSCommand, log id: 5549e288
>2012-12-17 08:17:37,467 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, ResetIrsVDSCommand, log id: 6e09901
>2012-12-17 08:17:37,478 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 508e5d12
>2012-12-17 08:17:37,478 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:17:37,478 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 508e5d12
>2012-12-17 08:17:37,535 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:17:37,561 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4495
>2012-12-17 08:17:37,562 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4495, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 69687062
>2012-12-17 08:17:37,585 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] spmStart polling started: taskId = 2e548bc5-8d22-4e18-aeb6-f153fc3276aa
>2012-12-17 08:17:38,607 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] spmStart polling ended: taskId = 2e548bc5-8d22-4e18-aeb6-f153fc3276aa task status = finished
>2012-12-17 08:17:38,625 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] spmStart polling ended, spm status: SPM
>2012-12-17 08:17:38,627 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2e548bc5-8d22-4e18-aeb6-f153fc3276aa), log id: 5417c0fc
>2012-12-17 08:17:38,645 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, HSMClearTaskVDSCommand, log id: 5417c0fc
>2012-12-17 08:17:38,645 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@637f0b85, log id: 69687062
>2012-12-17 08:17:38,652 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:17:38,666 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:17:38,669 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 41233a21
>2012-12-17 08:17:38,692 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 41233a21
>2012-12-17 08:17:38,692 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 357be1d1
>2012-12-17 08:17:38,692 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [5a4d3cd3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:17:38,693 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1777a538
>2012-12-17 08:17:38,693 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:17:38,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3c9e319c
>2012-12-17 08:17:38,712 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3c9e319c
>2012-12-17 08:17:38,713 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1777a538
>2012-12-17 08:17:38,713 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [5a4d3cd3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:17:46,342 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:17:47,574 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [54d258ba] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:17:47,698 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [667bec9e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:17:47,757 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [667bec9e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5435fe04
>2012-12-17 08:17:47,759 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [667bec9e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3b062d28
>2012-12-17 08:17:47,779 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [667bec9e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:17:48,433 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [667bec9e] FINISH, SpmStopVDSCommand, log id: 3b062d28
>2012-12-17 08:17:48,451 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [667bec9e] FINISH, ResetIrsVDSCommand, log id: 5435fe04
>2012-12-17 08:17:48,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [667bec9e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1126be8a
>2012-12-17 08:17:48,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [667bec9e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:17:48,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [667bec9e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1126be8a
>2012-12-17 08:17:56,572 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [69560fec] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:17:58,519 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [60181568] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:17:58,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [60181568] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4496
>2012-12-17 08:17:58,532 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [60181568] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4496, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 47052de0
>2012-12-17 08:17:58,555 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [60181568] spmStart polling started: taskId = d30aa432-6e3a-4984-b1b2-7ce87cdd91cb
>2012-12-17 08:17:59,578 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [60181568] spmStart polling ended: taskId = d30aa432-6e3a-4984-b1b2-7ce87cdd91cb task status = finished
>2012-12-17 08:17:59,596 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [60181568] spmStart polling ended, spm status: SPM
>2012-12-17 08:17:59,597 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [60181568] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d30aa432-6e3a-4984-b1b2-7ce87cdd91cb), log id: 494effba
>2012-12-17 08:17:59,616 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [60181568] FINISH, HSMClearTaskVDSCommand, log id: 494effba
>2012-12-17 08:17:59,616 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [60181568] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@617731ef, log id: 47052de0
>2012-12-17 08:17:59,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [60181568] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:17:59,824 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [1b451d62] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:17:59,824 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 73ef9045
>2012-12-17 08:17:59,834 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [60181568] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:17:59,975 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [36fcbeec] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:18:00,077 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 597519fd
>2012-12-17 08:18:00,079 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 772227bb
>2012-12-17 08:18:00,099 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:00,629 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] FINISH, SpmStopVDSCommand, log id: 772227bb
>2012-12-17 08:18:00,797 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] FINISH, ResetIrsVDSCommand, log id: 597519fd
>2012-12-17 08:18:00,808 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b4b4032
>2012-12-17 08:18:00,808 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [36fcbeec] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:00,808 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b4b4032
>2012-12-17 08:18:01,036 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1b451d62] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:18:01,048 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1b451d62] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4497
>2012-12-17 08:18:01,049 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4497, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 186189d8
>2012-12-17 08:18:01,072 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] spmStart polling started: taskId = 00044d67-86d7-4c1b-b721-39e201da45ec
>2012-12-17 08:18:02,099 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] spmStart polling ended: taskId = 00044d67-86d7-4c1b-b721-39e201da45ec task status = finished
>2012-12-17 08:18:02,117 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] spmStart polling ended, spm status: SPM
>2012-12-17 08:18:02,118 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=00044d67-86d7-4c1b-b721-39e201da45ec), log id: 7016235e
>2012-12-17 08:18:02,137 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, HSMClearTaskVDSCommand, log id: 7016235e
>2012-12-17 08:18:02,137 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3d6c29cf, log id: 186189d8
>2012-12-17 08:18:02,188 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1b451d62] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:18:02,237 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:02,239 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 326ee144
>2012-12-17 08:18:02,260 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 326ee144
>2012-12-17 08:18:02,260 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 73ef9045
>2012-12-17 08:18:02,260 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [1b451d62] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:18:02,261 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: d87513a
>2012-12-17 08:18:02,261 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:02,262 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: f352c8c
>2012-12-17 08:18:02,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: f352c8c
>2012-12-17 08:18:02,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: d87513a
>2012-12-17 08:18:02,281 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [1b451d62] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:18:06,794 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [6e8742f7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:18:11,102 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [54a63bb2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:18:11,274 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [323f195a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:18:11,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [323f195a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3930151f
>2012-12-17 08:18:11,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [323f195a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: ecc7ad7
>2012-12-17 08:18:11,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [323f195a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:11,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [323f195a] FINISH, SpmStopVDSCommand, log id: ecc7ad7
>2012-12-17 08:18:12,019 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [323f195a] FINISH, ResetIrsVDSCommand, log id: 3930151f
>2012-12-17 08:18:12,031 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [323f195a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6946e694
>2012-12-17 08:18:12,031 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [323f195a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:12,031 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [323f195a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6946e694
>2012-12-17 08:18:17,227 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [6ad64b5d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:18:22,079 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3725f22] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:18:22,114 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3725f22] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4498
>2012-12-17 08:18:22,116 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3725f22] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4498, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5469a2fc
>2012-12-17 08:18:22,139 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3725f22] spmStart polling started: taskId = 7b9358e5-3985-47c9-aed2-a8ec71a7592a
>2012-12-17 08:18:23,161 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3725f22] spmStart polling ended: taskId = 7b9358e5-3985-47c9-aed2-a8ec71a7592a task status = finished
>2012-12-17 08:18:23,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3725f22] spmStart polling ended, spm status: SPM
>2012-12-17 08:18:23,180 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [3725f22] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7b9358e5-3985-47c9-aed2-a8ec71a7592a), log id: 37d7e074
>2012-12-17 08:18:23,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [3725f22] FINISH, HSMClearTaskVDSCommand, log id: 37d7e074
>2012-12-17 08:18:23,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3725f22] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3dd0fc50, log id: 5469a2fc
>2012-12-17 08:18:23,234 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3725f22] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:18:23,285 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-11) [2d4b4fe5] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:18:23,285 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7eb9922a
>2012-12-17 08:18:23,320 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3725f22] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:18:23,403 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [252efffb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:18:23,449 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [252efffb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1b21a1ec
>2012-12-17 08:18:23,450 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [252efffb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 77a247be
>2012-12-17 08:18:23,471 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [252efffb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:23,622 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [252efffb] FINISH, SpmStopVDSCommand, log id: 77a247be
>2012-12-17 08:18:23,641 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [252efffb] FINISH, ResetIrsVDSCommand, log id: 1b21a1ec
>2012-12-17 08:18:23,652 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [252efffb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 19bc99cf
>2012-12-17 08:18:23,652 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [252efffb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:23,652 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [252efffb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 19bc99cf
>2012-12-17 08:18:23,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:18:23,712 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4499
>2012-12-17 08:18:23,714 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4499, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7eb16f28
>2012-12-17 08:18:23,737 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling started: taskId = dde5d09a-b08a-4a50-bb07-d77c5818bc70
>2012-12-17 08:18:24,759 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling ended: taskId = dde5d09a-b08a-4a50-bb07-d77c5818bc70 task status = finished
>2012-12-17 08:18:24,778 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling ended, spm status: SPM
>2012-12-17 08:18:24,779 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dde5d09a-b08a-4a50-bb07-d77c5818bc70), log id: 1da8a934
>2012-12-17 08:18:24,798 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, HSMClearTaskVDSCommand, log id: 1da8a934
>2012-12-17 08:18:24,798 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2bfca422, log id: 7eb16f28
>2012-12-17 08:18:24,810 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:18:24,822 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:24,827 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2874d83b
>2012-12-17 08:18:24,846 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2874d83b
>2012-12-17 08:18:24,846 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7eb9922a
>2012-12-17 08:18:24,847 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-11) [2d4b4fe5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:18:24,847 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5a13e8d
>2012-12-17 08:18:24,847 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:24,848 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 19c68305
>2012-12-17 08:18:24,867 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 19c68305
>2012-12-17 08:18:24,867 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5a13e8d
>2012-12-17 08:18:24,867 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-11) [2d4b4fe5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:18:27,427 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [42c50c6a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:18:33,728 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [4b0ad6c8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:18:33,851 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-22) [48aa9372] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:18:33,924 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 44bfcc46
>2012-12-17 08:18:33,926 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: a7cd5df
>2012-12-17 08:18:33,947 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:34,128 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] FINISH, SpmStopVDSCommand, log id: a7cd5df
>2012-12-17 08:18:34,144 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] FINISH, ResetIrsVDSCommand, log id: 44bfcc46
>2012-12-17 08:18:34,156 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 324d359a
>2012-12-17 08:18:34,156 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [48aa9372] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:34,156 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 324d359a
>2012-12-17 08:18:37,623 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [4f7e897c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:18:44,252 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [69560fec] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:18:44,331 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [69560fec] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4500
>2012-12-17 08:18:44,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [69560fec] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4500, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4b72a5b
>2012-12-17 08:18:44,358 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [69560fec] spmStart polling started: taskId = 77afb68e-2130-49e9-b91f-92d4dfb55140
>2012-12-17 08:18:45,380 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [69560fec] spmStart polling ended: taskId = 77afb68e-2130-49e9-b91f-92d4dfb55140 task status = finished
>2012-12-17 08:18:45,398 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [69560fec] spmStart polling ended, spm status: SPM
>2012-12-17 08:18:45,400 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [69560fec] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=77afb68e-2130-49e9-b91f-92d4dfb55140), log id: 6b59fb6f
>2012-12-17 08:18:45,422 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [69560fec] FINISH, HSMClearTaskVDSCommand, log id: 6b59fb6f
>2012-12-17 08:18:45,422 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [69560fec] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2db5d1ea, log id: 4b72a5b
>2012-12-17 08:18:45,713 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [69560fec] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:18:45,888 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [8c4c643] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:18:45,888 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 569151c7
>2012-12-17 08:18:45,905 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [69560fec] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:18:45,998 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [bb53630] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:18:46,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [bb53630] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 195e4cb
>2012-12-17 08:18:46,038 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [bb53630] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 730a7c60
>2012-12-17 08:18:46,058 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [bb53630] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:46,234 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [bb53630] FINISH, SpmStopVDSCommand, log id: 730a7c60
>2012-12-17 08:18:46,254 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [bb53630] FINISH, ResetIrsVDSCommand, log id: 195e4cb
>2012-12-17 08:18:46,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [bb53630] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c64c6fd
>2012-12-17 08:18:46,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [bb53630] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:46,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [bb53630] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c64c6fd
>2012-12-17 08:18:46,334 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [8c4c643] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:18:46,347 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [8c4c643] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4501
>2012-12-17 08:18:46,348 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4501, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 65d4d291
>2012-12-17 08:18:46,370 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] spmStart polling started: taskId = 1dd19294-b642-4a31-b11a-77ec6642cb44
>2012-12-17 08:18:47,391 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] spmStart polling ended: taskId = 1dd19294-b642-4a31-b11a-77ec6642cb44 task status = finished
>2012-12-17 08:18:47,410 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] spmStart polling ended, spm status: SPM
>2012-12-17 08:18:47,411 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1dd19294-b642-4a31-b11a-77ec6642cb44), log id: 2b9e2bf2
>2012-12-17 08:18:47,431 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, HSMClearTaskVDSCommand, log id: 2b9e2bf2
>2012-12-17 08:18:47,431 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6d950010, log id: 65d4d291
>2012-12-17 08:18:47,444 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [8c4c643] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:18:47,481 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:47,485 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 48f3506a
>2012-12-17 08:18:47,504 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 48f3506a
>2012-12-17 08:18:47,504 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 569151c7
>2012-12-17 08:18:47,504 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [8c4c643] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:18:47,504 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2f7b3a39
>2012-12-17 08:18:47,504 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:47,506 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2b958633
>2012-12-17 08:18:47,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2b958633
>2012-12-17 08:18:47,524 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2f7b3a39
>2012-12-17 08:18:47,524 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [8c4c643] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:18:47,798 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [48110f6e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:18:56,371 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [57ef526a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:18:56,452 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [376f3052] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:18:56,523 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [376f3052] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: e8ff725
>2012-12-17 08:18:56,525 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [376f3052] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 52fcdc9b
>2012-12-17 08:18:56,545 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [376f3052] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:56,759 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [376f3052] FINISH, SpmStopVDSCommand, log id: 52fcdc9b
>2012-12-17 08:18:56,772 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [376f3052] FINISH, ResetIrsVDSCommand, log id: e8ff725
>2012-12-17 08:18:56,782 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [376f3052] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2a50e390
>2012-12-17 08:18:56,782 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [376f3052] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:18:56,782 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [376f3052] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2a50e390
>2012-12-17 08:18:58,072 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [61c8d0e2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:19:06,836 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [4e011c57] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:19:06,848 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [4e011c57] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4502
>2012-12-17 08:19:06,849 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [4e011c57] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4502, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 372cae3f
>2012-12-17 08:19:06,871 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [4e011c57] spmStart polling started: taskId = 799fc251-db3e-4282-be16-5e2692247abf
>2012-12-17 08:19:07,894 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [4e011c57] spmStart polling ended: taskId = 799fc251-db3e-4282-be16-5e2692247abf task status = finished
>2012-12-17 08:19:07,916 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [4e011c57] spmStart polling ended, spm status: SPM
>2012-12-17 08:19:07,918 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [4e011c57] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=799fc251-db3e-4282-be16-5e2692247abf), log id: 78ae1939
>2012-12-17 08:19:07,937 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [4e011c57] FINISH, HSMClearTaskVDSCommand, log id: 78ae1939
>2012-12-17 08:19:07,938 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [4e011c57] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6dfa2b14, log id: 372cae3f
>2012-12-17 08:19:07,950 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [4e011c57] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:19:07,982 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5b39276a] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:19:07,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3e71d21c
>2012-12-17 08:19:07,997 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [4e011c57] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:19:08,123 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:19:08,177 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: ceb3778
>2012-12-17 08:19:08,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 54a3aafd
>2012-12-17 08:19:08,212 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:08,299 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:19:08,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SpmStopVDSCommand, log id: 54a3aafd
>2012-12-17 08:19:08,382 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, ResetIrsVDSCommand, log id: ceb3778
>2012-12-17 08:19:08,393 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e73d544
>2012-12-17 08:19:08,393 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:08,393 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e73d544
>2012-12-17 08:19:08,434 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5b39276a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:19:08,442 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5b39276a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4503
>2012-12-17 08:19:08,443 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4503, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6067309e
>2012-12-17 08:19:08,466 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] spmStart polling started: taskId = 7e795a91-e38f-42ee-9fbb-28c4d9e56bba
>2012-12-17 08:19:09,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] spmStart polling ended: taskId = 7e795a91-e38f-42ee-9fbb-28c4d9e56bba task status = finished
>2012-12-17 08:19:09,508 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] spmStart polling ended, spm status: SPM
>2012-12-17 08:19:09,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7e795a91-e38f-42ee-9fbb-28c4d9e56bba), log id: 767854a6
>2012-12-17 08:19:09,528 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, HSMClearTaskVDSCommand, log id: 767854a6
>2012-12-17 08:19:09,528 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@55fd9b0d, log id: 6067309e
>2012-12-17 08:19:09,539 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5b39276a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:19:09,552 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:09,553 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d11f210
>2012-12-17 08:19:09,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d11f210
>2012-12-17 08:19:09,576 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3e71d21c
>2012-12-17 08:19:09,576 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5b39276a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:19:09,576 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 585d8b6a
>2012-12-17 08:19:09,576 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:09,578 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 21a260bb
>2012-12-17 08:19:09,580 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:19:09,580 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4cfa225a
>2012-12-17 08:19:09,613 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 21a260bb
>2012-12-17 08:19:09,613 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 585d8b6a
>2012-12-17 08:19:09,613 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5b39276a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:19:09,613 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:09,616 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 51aa27d8
>2012-12-17 08:19:09,636 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 51aa27d8
>2012-12-17 08:19:09,636 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4cfa225a
>2012-12-17 08:19:09,636 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:19:09,636 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4d2f4495
>2012-12-17 08:19:09,636 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:09,637 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5b8ef8d
>2012-12-17 08:19:09,656 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5b8ef8d
>2012-12-17 08:19:09,656 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4d2f4495
>2012-12-17 08:19:09,656 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:19:18,499 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [55cb11f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:19:18,582 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [53e8b92c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:19:18,636 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 63b0d77d
>2012-12-17 08:19:18,638 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 486ad17a
>2012-12-17 08:19:18,657 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:18,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] FINISH, SpmStopVDSCommand, log id: 486ad17a
>2012-12-17 08:19:18,836 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] FINISH, ResetIrsVDSCommand, log id: 63b0d77d
>2012-12-17 08:19:18,847 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 416bd07e
>2012-12-17 08:19:18,847 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [53e8b92c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:18,847 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 416bd07e
>2012-12-17 08:19:28,671 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:19:28,890 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [7d2c68ac] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:19:28,911 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [7d2c68ac] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4504
>2012-12-17 08:19:28,922 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [7d2c68ac] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4504, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5542d527
>2012-12-17 08:19:28,944 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [7d2c68ac] spmStart polling started: taskId = c1d4d70f-ad84-4b12-a6fa-a69438848a28
>2012-12-17 08:19:30,055 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [7d2c68ac] spmStart polling ended: taskId = c1d4d70f-ad84-4b12-a6fa-a69438848a28 task status = finished
>2012-12-17 08:19:30,074 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [7d2c68ac] spmStart polling ended, spm status: SPM
>2012-12-17 08:19:30,075 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [7d2c68ac] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c1d4d70f-ad84-4b12-a6fa-a69438848a28), log id: 23550266
>2012-12-17 08:19:30,098 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [7d2c68ac] FINISH, HSMClearTaskVDSCommand, log id: 23550266
>2012-12-17 08:19:30,098 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [7d2c68ac] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@189b2c00, log id: 5542d527
>2012-12-17 08:19:30,131 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [7d2c68ac] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:19:31,343 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [55210d2] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:19:31,344 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7467f20f
>2012-12-17 08:19:31,344 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [7d2c68ac] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:19:31,893 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [76a14bbb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:19:31,958 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: fc5e015
>2012-12-17 08:19:31,960 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 635b13a8
>2012-12-17 08:19:31,980 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:32,143 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, SpmStopVDSCommand, log id: 635b13a8
>2012-12-17 08:19:32,184 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, ResetIrsVDSCommand, log id: fc5e015
>2012-12-17 08:19:32,195 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4d6c751b
>2012-12-17 08:19:32,195 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [76a14bbb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:32,195 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4d6c751b
>2012-12-17 08:19:32,257 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [55210d2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:19:32,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [55210d2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4505
>2012-12-17 08:19:32,271 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [55210d2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4505, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7e78bcb4
>2012-12-17 08:19:32,292 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [55210d2] spmStart polling started: taskId = edaf8c0a-b100-44ad-b86c-216eeb97a6da
>2012-12-17 08:19:33,313 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [55210d2] spmStart polling ended: taskId = edaf8c0a-b100-44ad-b86c-216eeb97a6da task status = finished
>2012-12-17 08:19:33,332 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [55210d2] spmStart polling ended, spm status: SPM
>2012-12-17 08:19:33,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [55210d2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=edaf8c0a-b100-44ad-b86c-216eeb97a6da), log id: 159d5ba1
>2012-12-17 08:19:33,351 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [55210d2] FINISH, HSMClearTaskVDSCommand, log id: 159d5ba1
>2012-12-17 08:19:33,351 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [55210d2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@deab21d, log id: 7e78bcb4
>2012-12-17 08:19:33,361 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [55210d2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:19:33,372 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:33,376 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 528c8c1
>2012-12-17 08:19:33,399 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 528c8c1
>2012-12-17 08:19:33,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7467f20f
>2012-12-17 08:19:33,400 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [55210d2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:19:33,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 50b480bb
>2012-12-17 08:19:33,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:33,401 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4cb19ea8
>2012-12-17 08:19:33,419 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4cb19ea8
>2012-12-17 08:19:33,419 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 50b480bb
>2012-12-17 08:19:33,419 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [55210d2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:19:38,861 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:19:42,299 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [780d3d0a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:19:42,391 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [2e21b599] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:19:42,440 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7e279467
>2012-12-17 08:19:42,442 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1d55106a
>2012-12-17 08:19:42,462 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:42,654 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, SpmStopVDSCommand, log id: 1d55106a
>2012-12-17 08:19:42,668 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, ResetIrsVDSCommand, log id: 7e279467
>2012-12-17 08:19:42,679 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1b790c61
>2012-12-17 08:19:42,679 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:42,679 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1b790c61
>2012-12-17 08:19:49,116 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [508b37e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:19:52,727 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [672e17aa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:19:52,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [672e17aa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4506
>2012-12-17 08:19:52,743 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [672e17aa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4506, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 665d1725
>2012-12-17 08:19:52,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [672e17aa] spmStart polling started: taskId = a265ba60-0a3e-4c66-a952-090480a0a481
>2012-12-17 08:19:53,788 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [672e17aa] spmStart polling ended: taskId = a265ba60-0a3e-4c66-a952-090480a0a481 task status = finished
>2012-12-17 08:19:53,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [672e17aa] spmStart polling ended, spm status: SPM
>2012-12-17 08:19:53,810 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [672e17aa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a265ba60-0a3e-4c66-a952-090480a0a481), log id: 198b15a0
>2012-12-17 08:19:53,829 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [672e17aa] FINISH, HSMClearTaskVDSCommand, log id: 198b15a0
>2012-12-17 08:19:53,829 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [672e17aa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7a38c879, log id: 665d1725
>2012-12-17 08:19:53,842 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [672e17aa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:19:53,878 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [55d82264] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:19:53,878 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 35f11e43
>2012-12-17 08:19:53,898 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [672e17aa] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:19:54,001 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [6bdbfd31] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:19:54,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2dd81042
>2012-12-17 08:19:54,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 723a9f9a
>2012-12-17 08:19:54,072 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:54,264 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, SpmStopVDSCommand, log id: 723a9f9a
>2012-12-17 08:19:54,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, ResetIrsVDSCommand, log id: 2dd81042
>2012-12-17 08:19:54,292 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 42432b10
>2012-12-17 08:19:54,292 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [6bdbfd31] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:54,292 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 42432b10
>2012-12-17 08:19:54,337 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:19:54,356 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4507
>2012-12-17 08:19:54,357 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4507, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 91f4e0f
>2012-12-17 08:19:54,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling started: taskId = bb045fbf-b8c8-4489-8548-305a7d43608f
>2012-12-17 08:19:55,403 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling ended: taskId = bb045fbf-b8c8-4489-8548-305a7d43608f task status = finished
>2012-12-17 08:19:55,422 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling ended, spm status: SPM
>2012-12-17 08:19:55,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bb045fbf-b8c8-4489-8548-305a7d43608f), log id: 487d1d7c
>2012-12-17 08:19:55,443 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, HSMClearTaskVDSCommand, log id: 487d1d7c
>2012-12-17 08:19:55,443 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e1add12, log id: 91f4e0f
>2012-12-17 08:19:55,784 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:19:55,898 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:55,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25bd1346
>2012-12-17 08:19:55,921 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25bd1346
>2012-12-17 08:19:55,921 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 35f11e43
>2012-12-17 08:19:55,921 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [55d82264] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:19:55,921 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 203c6c05
>2012-12-17 08:19:55,921 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:19:55,922 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3e5d9777
>2012-12-17 08:19:55,942 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3e5d9777
>2012-12-17 08:19:55,942 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 203c6c05
>2012-12-17 08:19:55,942 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [55d82264] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:19:59,292 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [267551eb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:20:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [5a4d3cd3] Checking autorecoverable hosts
>2012-12-17 08:20:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [5a4d3cd3] Autorecovering 0 hosts
>2012-12-17 08:20:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [5a4d3cd3] Checking autorecoverable hosts done
>2012-12-17 08:20:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [5a4d3cd3] Autorecovering storage domains is disabled, skipping
>2012-12-17 08:20:04,372 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [284eeca1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:20:04,485 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [2afeec77] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:20:04,555 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [2afeec77] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5c93c2c5
>2012-12-17 08:20:04,557 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [2afeec77] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 582f33ac
>2012-12-17 08:20:04,577 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [2afeec77] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:20:04,718 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [2afeec77] FINISH, SpmStopVDSCommand, log id: 582f33ac
>2012-12-17 08:20:04,766 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [2afeec77] FINISH, ResetIrsVDSCommand, log id: 5c93c2c5
>2012-12-17 08:20:04,776 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [2afeec77] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49140809
>2012-12-17 08:20:04,776 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [2afeec77] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:20:04,777 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [2afeec77] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49140809
>2012-12-17 08:20:09,518 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [48aa9372] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:20:14,834 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [6c123ee7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:20:14,847 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [6c123ee7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4508
>2012-12-17 08:20:14,849 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [6c123ee7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4508, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62b69b4
>2012-12-17 08:20:14,872 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [6c123ee7] spmStart polling started: taskId = 11dfe620-2798-4dd3-81ff-548a07d8465c
>2012-12-17 08:20:15,894 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [6c123ee7] spmStart polling ended: taskId = 11dfe620-2798-4dd3-81ff-548a07d8465c task status = finished
>2012-12-17 08:20:15,913 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [6c123ee7] spmStart polling ended, spm status: SPM
>2012-12-17 08:20:15,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [6c123ee7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=11dfe620-2798-4dd3-81ff-548a07d8465c), log id: 6caf23f3
>2012-12-17 08:20:15,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [6c123ee7] FINISH, HSMClearTaskVDSCommand, log id: 6caf23f3
>2012-12-17 08:20:15,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [6c123ee7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5e615e43, log id: 62b69b4
>2012-12-17 08:20:15,946 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [6c123ee7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:20:15,983 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [25ec3656] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:20:15,984 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7bc476ac
>2012-12-17 08:20:16,006 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [6c123ee7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:20:16,100 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-20) [cdacbd1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:20:16,145 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5c86030c
>2012-12-17 08:20:16,146 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5dd4de77
>2012-12-17 08:20:16,166 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:20:16,648 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, SpmStopVDSCommand, log id: 5dd4de77
>2012-12-17 08:20:16,660 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, ResetIrsVDSCommand, log id: 5c86030c
>2012-12-17 08:20:16,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7b6f69fe
>2012-12-17 08:20:16,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:20:16,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7b6f69fe
>2012-12-17 08:20:16,729 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:20:16,739 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4509
>2012-12-17 08:20:16,740 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4509, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2583dad2
>2012-12-17 08:20:16,763 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] spmStart polling started: taskId = 8da204d2-ad34-4ee7-8ec6-e6f3ff7feb27
>2012-12-17 08:20:17,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] spmStart polling ended: taskId = 8da204d2-ad34-4ee7-8ec6-e6f3ff7feb27 task status = finished
>2012-12-17 08:20:17,802 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] spmStart polling ended, spm status: SPM
>2012-12-17 08:20:17,804 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8da204d2-ad34-4ee7-8ec6-e6f3ff7feb27), log id: 13f5a565
>2012-12-17 08:20:17,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, HSMClearTaskVDSCommand, log id: 13f5a565
>2012-12-17 08:20:17,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@643d941f, log id: 2583dad2
>2012-12-17 08:20:17,839 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:20:17,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:20:17,862 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 260fb82e
>2012-12-17 08:20:17,881 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 260fb82e
>2012-12-17 08:20:17,881 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7bc476ac
>2012-12-17 08:20:17,881 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [25ec3656] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:20:17,881 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5204e787
>2012-12-17 08:20:17,881 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:20:17,882 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 41d4e89
>2012-12-17 08:20:17,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 41d4e89
>2012-12-17 08:20:17,901 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5204e787
>2012-12-17 08:20:17,901 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [25ec3656] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:20:19,767 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1879810a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:20:26,765 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6123283f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:20:26,993 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-45) [6e9d510a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:20:27,072 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 49805ebe
>2012-12-17 08:20:27,073 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4d83d4cf
>2012-12-17 08:20:27,096 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:20:27,266 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, SpmStopVDSCommand, log id: 4d83d4cf
>2012-12-17 08:20:27,278 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, ResetIrsVDSCommand, log id: 49805ebe
>2012-12-17 08:20:27,289 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 422419f9
>2012-12-17 08:20:27,289 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:20:27,289 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 422419f9
>2012-12-17 08:20:29,938 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [2e10036] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:20:37,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [508b37e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:20:37,408 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [508b37e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4510
>2012-12-17 08:20:37,410 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [508b37e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4510, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1fe4cff7
>2012-12-17 08:20:37,433 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [508b37e] spmStart polling started: taskId = 6f8cd004-c2f8-4a5b-9634-3024632a0b81
>2012-12-17 08:20:39,482 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [508b37e] spmStart polling ended: taskId = 6f8cd004-c2f8-4a5b-9634-3024632a0b81 task status = finished
>2012-12-17 08:20:39,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [508b37e] spmStart polling ended, spm status: SPM
>2012-12-17 08:20:39,502 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [508b37e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6f8cd004-c2f8-4a5b-9634-3024632a0b81), log id: 633661c7
>2012-12-17 08:20:39,520 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [508b37e] FINISH, HSMClearTaskVDSCommand, log id: 633661c7
>2012-12-17 08:20:39,520 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [508b37e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@74d82b4, log id: 1fe4cff7
>2012-12-17 08:20:39,528 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [508b37e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:20:39,561 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [555e5948] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:20:39,561 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6896289e
>2012-12-17 08:20:39,573 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [508b37e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:20:39,725 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [3ae71176] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:20:39,810 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 12ccb585
>2012-12-17 08:20:39,811 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 53bcdb57
>2012-12-17 08:20:39,831 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:20:39,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] FINISH, SpmStopVDSCommand, log id: 53bcdb57
>2012-12-17 08:20:39,988 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] FINISH, ResetIrsVDSCommand, log id: 12ccb585
>2012-12-17 08:20:39,999 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1bfaad51
>2012-12-17 08:20:39,999 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3ae71176] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:20:39,999 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1bfaad51
>2012-12-17 08:20:40,055 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [555e5948] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:20:40,076 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [555e5948] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4511
>2012-12-17 08:20:40,078 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4511, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 303165ae
>2012-12-17 08:20:40,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] spmStart polling started: taskId = c93e1d07-c44f-4931-b41b-11ec50f680d3
>2012-12-17 08:20:41,127 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] spmStart polling ended: taskId = c93e1d07-c44f-4931-b41b-11ec50f680d3 task status = finished
>2012-12-17 08:20:41,145 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] spmStart polling ended, spm status: SPM
>2012-12-17 08:20:41,146 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c93e1d07-c44f-4931-b41b-11ec50f680d3), log id: 6d721fbc
>2012-12-17 08:20:41,164 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, HSMClearTaskVDSCommand, log id: 6d721fbc
>2012-12-17 08:20:41,164 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5f81a7c6, log id: 303165ae
>2012-12-17 08:20:41,320 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [555e5948] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:20:41,353 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:20:41,355 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: de7e2b8
>2012-12-17 08:20:41,374 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: de7e2b8
>2012-12-17 08:20:41,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6896289e
>2012-12-17 08:20:41,374 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [555e5948] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:20:41,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 72d5a529
>2012-12-17 08:20:41,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:20:41,376 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 19a8d474
>2012-12-17 08:20:41,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 19a8d474
>2012-12-17 08:20:41,395 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 72d5a529
>2012-12-17 08:20:41,395 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [555e5948] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:20:50,089 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [42c50c6a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:20:51,477 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [3116bd71] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:20:51,514 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-43) [7582e2ab] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:20:51,568 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7de310d7
>2012-12-17 08:20:51,570 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3a56a777
>2012-12-17 08:20:51,590 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:20:51,748 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SpmStopVDSCommand, log id: 3a56a777
>2012-12-17 08:20:51,772 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, ResetIrsVDSCommand, log id: 7de310d7
>2012-12-17 08:20:51,847 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 25ec8691
>2012-12-17 08:20:51,847 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:20:51,847 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 25ec8691
>2012-12-17 08:21:01,906 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [16233e3c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:21:02,283 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [16233e3c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4512
>2012-12-17 08:21:02,285 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4512, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 38960148
>2012-12-17 08:21:02,308 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] spmStart polling started: taskId = f13a806d-10f4-4625-961d-b64eba34017e
>2012-12-17 08:21:03,330 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] spmStart polling ended: taskId = f13a806d-10f4-4625-961d-b64eba34017e task status = finished
>2012-12-17 08:21:03,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] spmStart polling ended, spm status: SPM
>2012-12-17 08:21:03,350 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f13a806d-10f4-4625-961d-b64eba34017e), log id: 2c39f0c4
>2012-12-17 08:21:03,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] FINISH, HSMClearTaskVDSCommand, log id: 2c39f0c4
>2012-12-17 08:21:03,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@fedea8b, log id: 38960148
>2012-12-17 08:21:03,378 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [16233e3c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:21:03,485 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [16233e3c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:21:03,648 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [42bc2eb6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:21:03,725 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [42bc2eb6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 60342cfd
>2012-12-17 08:21:03,726 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [42bc2eb6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 66112860
>2012-12-17 08:21:03,746 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [42bc2eb6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:21:03,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [42bc2eb6] FINISH, SpmStopVDSCommand, log id: 66112860
>2012-12-17 08:21:03,899 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [42bc2eb6] FINISH, ResetIrsVDSCommand, log id: 60342cfd
>2012-12-17 08:21:03,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [42bc2eb6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 75f8fa48
>2012-12-17 08:21:03,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [42bc2eb6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:21:03,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [42bc2eb6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 75f8fa48
>2012-12-17 08:21:12,570 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [36fcbeec] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:21:13,975 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1b451d62] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:21:14,006 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1b451d62] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4513
>2012-12-17 08:21:14,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4513, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7968eb0f
>2012-12-17 08:21:14,029 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] spmStart polling started: taskId = 09f13bd1-7812-4e76-874d-5fd46ad2cc87
>2012-12-17 08:21:15,050 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] spmStart polling ended: taskId = 09f13bd1-7812-4e76-874d-5fd46ad2cc87 task status = finished
>2012-12-17 08:21:15,070 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] spmStart polling ended, spm status: SPM
>2012-12-17 08:21:15,072 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=09f13bd1-7812-4e76-874d-5fd46ad2cc87), log id: 4d0cc38a
>2012-12-17 08:21:15,090 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, HSMClearTaskVDSCommand, log id: 4d0cc38a
>2012-12-17 08:21:15,090 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@f9c08b, log id: 7968eb0f
>2012-12-17 08:21:15,100 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1b451d62] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:21:15,136 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [6e9d510a] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:21:15,136 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2eb08483
>2012-12-17 08:21:15,159 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1b451d62] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:21:16,235 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [7bc37769] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:21:16,312 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2dd6cf7d
>2012-12-17 08:21:16,313 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f79fd76
>2012-12-17 08:21:16,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:21:16,666 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] FINISH, SpmStopVDSCommand, log id: 4f79fd76
>2012-12-17 08:21:16,682 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] FINISH, ResetIrsVDSCommand, log id: 2dd6cf7d
>2012-12-17 08:21:16,693 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 63f28439
>2012-12-17 08:21:16,693 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [7bc37769] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:21:16,693 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 63f28439
>2012-12-17 08:21:16,752 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:21:16,762 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4514
>2012-12-17 08:21:16,763 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4514, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e5636b4
>2012-12-17 08:21:16,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] spmStart polling started: taskId = 156fb605-d70d-4649-9872-7123dd9a2c6b
>2012-12-17 08:21:17,808 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] spmStart polling ended: taskId = 156fb605-d70d-4649-9872-7123dd9a2c6b task status = finished
>2012-12-17 08:21:17,828 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] spmStart polling ended, spm status: SPM
>2012-12-17 08:21:17,829 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=156fb605-d70d-4649-9872-7123dd9a2c6b), log id: 5cc6900c
>2012-12-17 08:21:17,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, HSMClearTaskVDSCommand, log id: 5cc6900c
>2012-12-17 08:21:17,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@510dca02, log id: 5e5636b4
>2012-12-17 08:21:17,858 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:21:17,873 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:21:17,874 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 50d0c5e4
>2012-12-17 08:21:17,897 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 50d0c5e4
>2012-12-17 08:21:17,897 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2eb08483
>2012-12-17 08:21:17,897 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [6e9d510a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:21:17,898 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 42cc0ff9
>2012-12-17 08:21:17,898 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:21:17,899 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1f7cbbc3
>2012-12-17 08:21:17,918 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1f7cbbc3
>2012-12-17 08:21:17,918 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 42cc0ff9
>2012-12-17 08:21:17,918 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [6e9d510a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:21:22,766 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:21:26,804 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [6ad64b5d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:21:26,964 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [1852110b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:21:27,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [1852110b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7f7dfad1
>2012-12-17 08:21:27,038 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [1852110b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3ad24ff7
>2012-12-17 08:21:27,059 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [1852110b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:21:27,671 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [1852110b] FINISH, SpmStopVDSCommand, log id: 3ad24ff7
>2012-12-17 08:21:27,707 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [1852110b] FINISH, ResetIrsVDSCommand, log id: 7f7dfad1
>2012-12-17 08:21:27,719 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [1852110b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c053df8
>2012-12-17 08:21:27,719 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1852110b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:21:27,719 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [1852110b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c053df8
>2012-12-17 08:21:32,978 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:21:37,777 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:21:37,797 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4515
>2012-12-17 08:21:37,799 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4515, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 903f6df
>2012-12-17 08:21:37,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] spmStart polling started: taskId = 97bb6416-275f-422f-bf1e-90a9c732f39e
>2012-12-17 08:21:38,842 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] spmStart polling ended: taskId = 97bb6416-275f-422f-bf1e-90a9c732f39e task status = finished
>2012-12-17 08:21:38,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] spmStart polling ended, spm status: SPM
>2012-12-17 08:21:38,862 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=97bb6416-275f-422f-bf1e-90a9c732f39e), log id: 3f6680fd
>2012-12-17 08:21:38,880 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, HSMClearTaskVDSCommand, log id: 3f6680fd
>2012-12-17 08:21:38,880 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@629e828f, log id: 903f6df
>2012-12-17 08:21:38,912 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:21:38,974 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [7de89d45] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:21:38,974 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: e0d0279
>2012-12-17 08:21:38,994 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:21:39,127 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-80) [495d106] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:21:39,186 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 62a575d1
>2012-12-17 08:21:39,188 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2c177f6b
>2012-12-17 08:21:39,208 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [495d106] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:21:39,357 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, SpmStopVDSCommand, log id: 2c177f6b
>2012-12-17 08:21:39,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, ResetIrsVDSCommand, log id: 62a575d1
>2012-12-17 08:21:39,380 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2a6f3e4c
>2012-12-17 08:21:39,380 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [495d106] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:21:39,380 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2a6f3e4c
>2012-12-17 08:21:39,421 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7de89d45] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:21:39,434 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7de89d45] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4516
>2012-12-17 08:21:39,435 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4516, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: aa0426f
>2012-12-17 08:21:39,458 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] spmStart polling started: taskId = 1a3e6f5e-870f-4273-a826-915d53fb4180
>2012-12-17 08:21:40,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] spmStart polling ended: taskId = 1a3e6f5e-870f-4273-a826-915d53fb4180 task status = finished
>2012-12-17 08:21:40,499 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] spmStart polling ended, spm status: SPM
>2012-12-17 08:21:40,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1a3e6f5e-870f-4273-a826-915d53fb4180), log id: 7cc18446
>2012-12-17 08:21:40,523 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] FINISH, HSMClearTaskVDSCommand, log id: 7cc18446
>2012-12-17 08:21:40,523 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7fe0b098, log id: aa0426f
>2012-12-17 08:21:40,916 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7de89d45] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:21:41,365 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:21:41,369 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4a400844
>2012-12-17 08:21:41,390 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4a400844
>2012-12-17 08:21:41,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: e0d0279
>2012-12-17 08:21:41,390 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [7de89d45] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:21:41,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 31dece71
>2012-12-17 08:21:41,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:21:41,392 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6f6eeb1f
>2012-12-17 08:21:41,410 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f6eeb1f
>2012-12-17 08:21:41,410 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 31dece71
>2012-12-17 08:21:41,410 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [7de89d45] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:21:43,172 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [3e88c176] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:21:49,479 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [219a053f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:21:49,652 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [d425472] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:21:49,729 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [d425472] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7e87f2a7
>2012-12-17 08:21:49,731 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [d425472] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2e764c31
>2012-12-17 08:21:49,751 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [d425472] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:21:49,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [d425472] FINISH, SpmStopVDSCommand, log id: 2e764c31
>2012-12-17 08:21:49,924 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [d425472] FINISH, ResetIrsVDSCommand, log id: 7e87f2a7
>2012-12-17 08:21:49,935 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [d425472] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b42a617
>2012-12-17 08:21:49,935 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [d425472] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:21:49,935 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [d425472] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b42a617
>2012-12-17 08:21:53,357 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [381e4566] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:21:59,980 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [7a15027e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:21:59,995 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [7a15027e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4517
>2012-12-17 08:21:59,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7a15027e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4517, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5b4a7248
>2012-12-17 08:22:00,018 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7a15027e] spmStart polling started: taskId = b98c6a60-cdc1-4ed9-b7fc-10ab769441e3
>2012-12-17 08:22:01,041 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7a15027e] spmStart polling ended: taskId = b98c6a60-cdc1-4ed9-b7fc-10ab769441e3 task status = finished
>2012-12-17 08:22:01,059 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7a15027e] spmStart polling ended, spm status: SPM
>2012-12-17 08:22:01,060 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [7a15027e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b98c6a60-cdc1-4ed9-b7fc-10ab769441e3), log id: 29817262
>2012-12-17 08:22:01,079 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [7a15027e] FINISH, HSMClearTaskVDSCommand, log id: 29817262
>2012-12-17 08:22:01,079 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7a15027e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@717c9537, log id: 5b4a7248
>2012-12-17 08:22:01,355 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [7a15027e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:22:01,409 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [71ec072c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:22:01,410 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1ecbeb53
>2012-12-17 08:22:01,422 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [7a15027e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:22:01,501 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-33) [5970076c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:22:01,590 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-33) [5970076c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 65159224
>2012-12-17 08:22:01,592 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [5970076c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4a46f5e1
>2012-12-17 08:22:01,612 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [5970076c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:01,738 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [5970076c] FINISH, SpmStopVDSCommand, log id: 4a46f5e1
>2012-12-17 08:22:01,753 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-33) [5970076c] FINISH, ResetIrsVDSCommand, log id: 65159224
>2012-12-17 08:22:01,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-33) [5970076c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c01c559
>2012-12-17 08:22:01,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [5970076c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:01,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-33) [5970076c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c01c559
>2012-12-17 08:22:01,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:22:01,848 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4518
>2012-12-17 08:22:01,849 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4518, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4523fb89
>2012-12-17 08:22:01,873 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] spmStart polling started: taskId = d1bc0697-1397-4a73-b9fd-bf57312cffc2
>2012-12-17 08:22:02,896 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] spmStart polling ended: taskId = d1bc0697-1397-4a73-b9fd-bf57312cffc2 task status = finished
>2012-12-17 08:22:02,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] spmStart polling ended, spm status: SPM
>2012-12-17 08:22:02,916 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d1bc0697-1397-4a73-b9fd-bf57312cffc2), log id: 55847e35
>2012-12-17 08:22:02,935 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, HSMClearTaskVDSCommand, log id: 55847e35
>2012-12-17 08:22:02,935 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6d79f56f, log id: 4523fb89
>2012-12-17 08:22:02,958 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:22:02,993 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:02,994 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 40de52ee
>2012-12-17 08:22:03,025 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 40de52ee
>2012-12-17 08:22:03,025 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1ecbeb53
>2012-12-17 08:22:03,025 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [71ec072c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:22:03,025 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 41b9a612
>2012-12-17 08:22:03,025 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:03,028 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1c33a992
>2012-12-17 08:22:03,047 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1c33a992
>2012-12-17 08:22:03,047 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 41b9a612
>2012-12-17 08:22:03,047 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [71ec072c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:22:03,572 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:22:11,873 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [f4188d3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:22:12,051 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [778f4477] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:22:12,153 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [778f4477] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1b860d90
>2012-12-17 08:22:12,155 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [778f4477] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3fae04cb
>2012-12-17 08:22:12,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [778f4477] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:12,738 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [778f4477] FINISH, SpmStopVDSCommand, log id: 3fae04cb
>2012-12-17 08:22:12,757 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [778f4477] FINISH, ResetIrsVDSCommand, log id: 1b860d90
>2012-12-17 08:22:12,768 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [778f4477] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 52f70926
>2012-12-17 08:22:12,768 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [778f4477] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:12,768 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [778f4477] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 52f70926
>2012-12-17 08:22:13,903 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [733ac637] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:22:22,837 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [252efffb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:22:22,867 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [252efffb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4519
>2012-12-17 08:22:22,868 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [252efffb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4519, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5dc7fbc5
>2012-12-17 08:22:22,890 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [252efffb] spmStart polling started: taskId = d8e84330-08fe-41e3-bcc4-f271934a58f3
>2012-12-17 08:22:23,912 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [252efffb] spmStart polling ended: taskId = d8e84330-08fe-41e3-bcc4-f271934a58f3 task status = finished
>2012-12-17 08:22:23,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [252efffb] spmStart polling ended, spm status: SPM
>2012-12-17 08:22:23,932 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [252efffb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d8e84330-08fe-41e3-bcc4-f271934a58f3), log id: 3c36b8eb
>2012-12-17 08:22:23,950 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [252efffb] FINISH, HSMClearTaskVDSCommand, log id: 3c36b8eb
>2012-12-17 08:22:23,950 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [252efffb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2ca66b2f, log id: 5dc7fbc5
>2012-12-17 08:22:23,988 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [252efffb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:22:24,073 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [43f46040] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:22:24,073 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 61abb047
>2012-12-17 08:22:24,096 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [252efffb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:22:24,239 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [24da4704] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:22:24,339 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [24da4704] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7a4f8fa4
>2012-12-17 08:22:24,341 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [24da4704] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3719d455
>2012-12-17 08:22:24,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [24da4704] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:24,556 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [24da4704] FINISH, SpmStopVDSCommand, log id: 3719d455
>2012-12-17 08:22:24,567 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [24da4704] FINISH, ResetIrsVDSCommand, log id: 7a4f8fa4
>2012-12-17 08:22:24,578 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [24da4704] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3843543e
>2012-12-17 08:22:24,578 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [24da4704] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:24,578 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [24da4704] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3843543e
>2012-12-17 08:22:24,638 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [43f46040] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:22:24,650 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [43f46040] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4520
>2012-12-17 08:22:24,651 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4520, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5b232817
>2012-12-17 08:22:24,675 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] spmStart polling started: taskId = adfea206-6d73-4cb9-b86a-e2aaba8016ec
>2012-12-17 08:22:25,697 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] spmStart polling ended: taskId = adfea206-6d73-4cb9-b86a-e2aaba8016ec task status = finished
>2012-12-17 08:22:25,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] spmStart polling ended, spm status: SPM
>2012-12-17 08:22:25,717 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=adfea206-6d73-4cb9-b86a-e2aaba8016ec), log id: 373b4882
>2012-12-17 08:22:25,736 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, HSMClearTaskVDSCommand, log id: 373b4882
>2012-12-17 08:22:25,736 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@15156b22, log id: 5b232817
>2012-12-17 08:22:25,798 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [43f46040] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:22:25,870 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:25,875 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 70926669
>2012-12-17 08:22:25,894 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 70926669
>2012-12-17 08:22:25,895 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 61abb047
>2012-12-17 08:22:25,895 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [43f46040] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:22:25,895 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5f9ae83f
>2012-12-17 08:22:25,895 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:25,896 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 53418b2
>2012-12-17 08:22:25,915 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 53418b2
>2012-12-17 08:22:25,915 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5f9ae83f
>2012-12-17 08:22:25,915 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [43f46040] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:22:34,306 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:22:34,681 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [667bec9e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:22:34,765 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [436410e6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:22:34,839 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 67d44a25
>2012-12-17 08:22:34,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 44976e8b
>2012-12-17 08:22:34,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [436410e6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:35,032 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, SpmStopVDSCommand, log id: 44976e8b
>2012-12-17 08:22:35,047 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, ResetIrsVDSCommand, log id: 67d44a25
>2012-12-17 08:22:35,057 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3e6d1939
>2012-12-17 08:22:35,057 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [436410e6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:35,057 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3e6d1939
>2012-12-17 08:22:44,532 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:22:45,139 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1879810a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:22:45,151 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1879810a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4521
>2012-12-17 08:22:45,152 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [1879810a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4521, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 136c5335
>2012-12-17 08:22:45,175 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [1879810a] spmStart polling started: taskId = a7da3510-78bc-44d2-8508-e195d3283129
>2012-12-17 08:22:46,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [1879810a] spmStart polling ended: taskId = a7da3510-78bc-44d2-8508-e195d3283129 task status = finished
>2012-12-17 08:22:46,216 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [1879810a] spmStart polling ended, spm status: SPM
>2012-12-17 08:22:46,217 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [1879810a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a7da3510-78bc-44d2-8508-e195d3283129), log id: 69273a85
>2012-12-17 08:22:46,236 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [1879810a] FINISH, HSMClearTaskVDSCommand, log id: 69273a85
>2012-12-17 08:22:46,236 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [1879810a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e7327b2, log id: 136c5335
>2012-12-17 08:22:46,361 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1879810a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:22:46,494 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) [7543fd1a] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:22:46,494 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3be7b4b8
>2012-12-17 08:22:46,518 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1879810a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:22:46,651 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [2c13a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:22:46,725 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [2c13a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4294af43
>2012-12-17 08:22:46,727 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [2c13a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1f8e1174
>2012-12-17 08:22:46,747 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [2c13a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:46,877 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [2c13a] FINISH, SpmStopVDSCommand, log id: 1f8e1174
>2012-12-17 08:22:46,888 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [2c13a] FINISH, ResetIrsVDSCommand, log id: 4294af43
>2012-12-17 08:22:46,899 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [2c13a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 663521de
>2012-12-17 08:22:46,899 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2c13a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:46,899 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [2c13a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 663521de
>2012-12-17 08:22:46,945 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7543fd1a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:22:46,961 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7543fd1a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4522
>2012-12-17 08:22:46,962 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4522, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 38b6b9b1
>2012-12-17 08:22:46,984 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] spmStart polling started: taskId = 38cbb08f-4377-489f-bff6-742897a2a3a4
>2012-12-17 08:22:48,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] spmStart polling ended: taskId = 38cbb08f-4377-489f-bff6-742897a2a3a4 task status = finished
>2012-12-17 08:22:48,026 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] spmStart polling ended, spm status: SPM
>2012-12-17 08:22:48,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=38cbb08f-4377-489f-bff6-742897a2a3a4), log id: 8ab4d1
>2012-12-17 08:22:48,045 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, HSMClearTaskVDSCommand, log id: 8ab4d1
>2012-12-17 08:22:48,045 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1265059e, log id: 38b6b9b1
>2012-12-17 08:22:48,059 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7543fd1a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:22:48,086 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:48,090 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 269af953
>2012-12-17 08:22:48,109 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 269af953
>2012-12-17 08:22:48,109 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3be7b4b8
>2012-12-17 08:22:48,109 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) [7543fd1a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:22:48,109 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 866cbf2
>2012-12-17 08:22:48,109 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:48,110 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5b7e58da
>2012-12-17 08:22:48,129 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5b7e58da
>2012-12-17 08:22:48,129 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 866cbf2
>2012-12-17 08:22:48,129 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) [7543fd1a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:22:54,740 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [376f3052] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:22:56,994 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [44853924] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:22:57,264 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-81) [3e5a5a03] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:22:57,367 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1effcdb1
>2012-12-17 08:22:57,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 41f10a40
>2012-12-17 08:22:57,388 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:57,591 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, SpmStopVDSCommand, log id: 41f10a40
>2012-12-17 08:22:57,607 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, ResetIrsVDSCommand, log id: 1effcdb1
>2012-12-17 08:22:57,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 8a6a68a
>2012-12-17 08:22:57,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:22:57,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 8a6a68a
>2012-12-17 08:23:04,958 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [6bdbfd31] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:23:07,665 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5b39276a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:23:07,673 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5b39276a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4523
>2012-12-17 08:23:07,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4523, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6cf31323
>2012-12-17 08:23:07,697 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] spmStart polling started: taskId = a0ef1178-c3fc-493d-afa6-e7fdfe352440
>2012-12-17 08:23:08,720 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] spmStart polling ended: taskId = a0ef1178-c3fc-493d-afa6-e7fdfe352440 task status = finished
>2012-12-17 08:23:08,738 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] spmStart polling ended, spm status: SPM
>2012-12-17 08:23:08,740 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a0ef1178-c3fc-493d-afa6-e7fdfe352440), log id: 603fb910
>2012-12-17 08:23:08,758 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, HSMClearTaskVDSCommand, log id: 603fb910
>2012-12-17 08:23:08,759 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@778efacd, log id: 6cf31323
>2012-12-17 08:23:08,766 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5b39276a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:23:08,831 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [1143d064] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:23:08,831 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 77017160
>2012-12-17 08:23:08,853 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5b39276a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:23:09,036 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [723e6c9b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:23:09,117 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [723e6c9b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5332a348
>2012-12-17 08:23:09,118 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [723e6c9b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6482688b
>2012-12-17 08:23:09,138 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [723e6c9b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:09,815 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [723e6c9b] FINISH, SpmStopVDSCommand, log id: 6482688b
>2012-12-17 08:23:09,827 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [723e6c9b] FINISH, ResetIrsVDSCommand, log id: 5332a348
>2012-12-17 08:23:09,839 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [723e6c9b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 453e65d5
>2012-12-17 08:23:09,839 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [723e6c9b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:09,839 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [723e6c9b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 453e65d5
>2012-12-17 08:23:09,876 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:23:09,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4524
>2012-12-17 08:23:09,895 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4524, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 303db02a
>2012-12-17 08:23:09,919 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] spmStart polling started: taskId = fe5a8e84-c373-42f3-aa6a-170cbc3bce31
>2012-12-17 08:23:10,941 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] spmStart polling ended: taskId = fe5a8e84-c373-42f3-aa6a-170cbc3bce31 task status = finished
>2012-12-17 08:23:10,960 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] spmStart polling ended, spm status: SPM
>2012-12-17 08:23:10,961 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=fe5a8e84-c373-42f3-aa6a-170cbc3bce31), log id: 3f0f366b
>2012-12-17 08:23:10,979 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, HSMClearTaskVDSCommand, log id: 3f0f366b
>2012-12-17 08:23:10,979 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7fbdcb24, log id: 303db02a
>2012-12-17 08:23:11,097 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:23:11,115 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:11,119 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6ab85557
>2012-12-17 08:23:11,138 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6ab85557
>2012-12-17 08:23:11,138 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 77017160
>2012-12-17 08:23:11,139 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [1143d064] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:23:11,139 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6beaf97d
>2012-12-17 08:23:11,139 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:11,140 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 49689b4
>2012-12-17 08:23:11,158 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 49689b4
>2012-12-17 08:23:11,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6beaf97d
>2012-12-17 08:23:11,158 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [1143d064] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:23:15,896 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [2afeec77] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:23:19,917 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [9dcd5cb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:23:20,019 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [43662b97] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:23:20,062 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [43662b97] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 538b4e92
>2012-12-17 08:23:20,063 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [43662b97] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4774a893
>2012-12-17 08:23:20,083 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [43662b97] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:20,241 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [43662b97] FINISH, SpmStopVDSCommand, log id: 4774a893
>2012-12-17 08:23:20,866 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [43662b97] FINISH, ResetIrsVDSCommand, log id: 538b4e92
>2012-12-17 08:23:20,876 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [43662b97] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 63e0791e
>2012-12-17 08:23:20,876 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [43662b97] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:20,876 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [43662b97] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 63e0791e
>2012-12-17 08:23:26,501 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6995ddd2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:23:31,305 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:23:31,314 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4525
>2012-12-17 08:23:31,315 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4525, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 11908b9
>2012-12-17 08:23:31,337 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] spmStart polling started: taskId = 7e4f0d30-d60b-4057-9e50-2c6109092f41
>2012-12-17 08:23:32,359 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] spmStart polling ended: taskId = 7e4f0d30-d60b-4057-9e50-2c6109092f41 task status = finished
>2012-12-17 08:23:32,377 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] spmStart polling ended, spm status: SPM
>2012-12-17 08:23:32,378 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7e4f0d30-d60b-4057-9e50-2c6109092f41), log id: 5d813997
>2012-12-17 08:23:32,397 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] FINISH, HSMClearTaskVDSCommand, log id: 5d813997
>2012-12-17 08:23:32,397 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4eec1876, log id: 11908b9
>2012-12-17 08:23:32,408 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:23:32,448 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [2f70811] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:23:32,448 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 589c6b47
>2012-12-17 08:23:32,468 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:23:32,608 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [50c114a6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:23:32,654 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [50c114a6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1a111bde
>2012-12-17 08:23:32,655 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [50c114a6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6401f4e3
>2012-12-17 08:23:32,675 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [50c114a6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:33,428 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [50c114a6] FINISH, SpmStopVDSCommand, log id: 6401f4e3
>2012-12-17 08:23:33,444 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [50c114a6] FINISH, ResetIrsVDSCommand, log id: 1a111bde
>2012-12-17 08:23:33,455 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [50c114a6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 61f0879b
>2012-12-17 08:23:33,455 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [50c114a6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:33,455 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [50c114a6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 61f0879b
>2012-12-17 08:23:33,513 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:23:33,526 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4526
>2012-12-17 08:23:33,527 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4526, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 39f86d1d
>2012-12-17 08:23:33,550 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] spmStart polling started: taskId = c6631b08-7a9f-4bbd-8753-8c670417cece
>2012-12-17 08:23:34,585 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] spmStart polling ended: taskId = c6631b08-7a9f-4bbd-8753-8c670417cece task status = finished
>2012-12-17 08:23:34,615 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] spmStart polling ended, spm status: SPM
>2012-12-17 08:23:34,617 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c6631b08-7a9f-4bbd-8753-8c670417cece), log id: 2814fab6
>2012-12-17 08:23:34,636 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, HSMClearTaskVDSCommand, log id: 2814fab6
>2012-12-17 08:23:34,636 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7957d748, log id: 39f86d1d
>2012-12-17 08:23:34,665 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:23:34,698 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:34,700 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 383b748
>2012-12-17 08:23:34,720 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 383b748
>2012-12-17 08:23:34,720 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 589c6b47
>2012-12-17 08:23:34,720 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [2f70811] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:23:34,720 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3c75d9c5
>2012-12-17 08:23:34,720 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:34,722 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4b5ee868
>2012-12-17 08:23:34,740 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b5ee868
>2012-12-17 08:23:34,740 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3c75d9c5
>2012-12-17 08:23:34,740 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [2f70811] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:23:36,820 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [7b78f9c5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:23:43,545 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [51190a74] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:23:43,638 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [6462735f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:23:43,683 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [6462735f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 699f1afc
>2012-12-17 08:23:43,685 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [6462735f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4a5022c4
>2012-12-17 08:23:43,705 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [6462735f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:43,852 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [6462735f] FINISH, SpmStopVDSCommand, log id: 4a5022c4
>2012-12-17 08:23:43,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [6462735f] FINISH, ResetIrsVDSCommand, log id: 699f1afc
>2012-12-17 08:23:43,905 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [6462735f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 157372b7
>2012-12-17 08:23:43,906 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [6462735f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:43,906 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [6462735f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 157372b7
>2012-12-17 08:23:47,005 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [323f195a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:23:53,953 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [555e5948] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:23:53,966 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [555e5948] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4527
>2012-12-17 08:23:53,968 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4527, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67a704ca
>2012-12-17 08:23:53,990 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] spmStart polling started: taskId = 20bb434f-039e-486a-b697-abd2cf23a7ea
>2012-12-17 08:23:55,013 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] spmStart polling ended: taskId = 20bb434f-039e-486a-b697-abd2cf23a7ea task status = finished
>2012-12-17 08:23:55,031 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] spmStart polling ended, spm status: SPM
>2012-12-17 08:23:55,032 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=20bb434f-039e-486a-b697-abd2cf23a7ea), log id: 16b8e7cf
>2012-12-17 08:23:55,050 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, HSMClearTaskVDSCommand, log id: 16b8e7cf
>2012-12-17 08:23:55,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@457ad2b0, log id: 67a704ca
>2012-12-17 08:23:55,062 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [555e5948] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:23:55,148 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [59932b3] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:23:55,148 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6d964b71
>2012-12-17 08:23:55,629 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [555e5948] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:23:56,342 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [3bf47b4b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:23:56,431 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [3bf47b4b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 62d5b49b
>2012-12-17 08:23:56,433 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [3bf47b4b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7807218c
>2012-12-17 08:23:56,453 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [3bf47b4b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:56,606 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [3bf47b4b] FINISH, SpmStopVDSCommand, log id: 7807218c
>2012-12-17 08:23:56,620 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [3bf47b4b] FINISH, ResetIrsVDSCommand, log id: 62d5b49b
>2012-12-17 08:23:56,632 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [3bf47b4b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ce9f4c3
>2012-12-17 08:23:56,632 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3bf47b4b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:56,632 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [3bf47b4b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ce9f4c3
>2012-12-17 08:23:56,727 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:23:56,776 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4528
>2012-12-17 08:23:56,777 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4528, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: eda47cb
>2012-12-17 08:23:56,800 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] spmStart polling started: taskId = fd130412-3c78-45f6-a333-8f6c3d90227f
>2012-12-17 08:23:57,821 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] spmStart polling ended: taskId = fd130412-3c78-45f6-a333-8f6c3d90227f task status = finished
>2012-12-17 08:23:57,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] spmStart polling ended, spm status: SPM
>2012-12-17 08:23:57,841 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=fd130412-3c78-45f6-a333-8f6c3d90227f), log id: 28d9ac6a
>2012-12-17 08:23:57,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, HSMClearTaskVDSCommand, log id: 28d9ac6a
>2012-12-17 08:23:57,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6ac67819, log id: eda47cb
>2012-12-17 08:23:57,891 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:23:57,947 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:57,952 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 71b7adeb
>2012-12-17 08:23:57,977 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 71b7adeb
>2012-12-17 08:23:57,977 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6d964b71
>2012-12-17 08:23:57,978 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [59932b3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:23:57,978 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e901274
>2012-12-17 08:23:57,978 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:23:57,980 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4d4377c0
>2012-12-17 08:23:58,021 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4d4377c0
>2012-12-17 08:23:58,021 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e901274
>2012-12-17 08:23:58,021 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [59932b3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:24:06,761 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [3e88c176] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:24:06,889 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [16c2b2c4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:24:06,937 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: f2c7639
>2012-12-17 08:24:06,939 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3c117069
>2012-12-17 08:24:06,958 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:24:07,109 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, SpmStopVDSCommand, log id: 3c117069
>2012-12-17 08:24:07,127 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, ResetIrsVDSCommand, log id: f2c7639
>2012-12-17 08:24:07,138 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3436f61f
>2012-12-17 08:24:07,138 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [16c2b2c4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:24:07,138 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3436f61f
>2012-12-17 08:24:07,472 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:24:17,198 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [33a3b92a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:24:17,211 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [33a3b92a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4529
>2012-12-17 08:24:17,213 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [33a3b92a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4529, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 583dcef7
>2012-12-17 08:24:17,234 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [33a3b92a] spmStart polling started: taskId = 8670f109-0a2a-43db-9616-56ce935f424d
>2012-12-17 08:24:18,255 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [33a3b92a] spmStart polling ended: taskId = 8670f109-0a2a-43db-9616-56ce935f424d task status = finished
>2012-12-17 08:24:18,273 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [33a3b92a] spmStart polling ended, spm status: SPM
>2012-12-17 08:24:18,274 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [33a3b92a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8670f109-0a2a-43db-9616-56ce935f424d), log id: 1e615734
>2012-12-17 08:24:18,292 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [33a3b92a] FINISH, HSMClearTaskVDSCommand, log id: 1e615734
>2012-12-17 08:24:18,292 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [33a3b92a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@8a42681, log id: 583dcef7
>2012-12-17 08:24:18,306 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [33a3b92a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:24:18,345 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [381e4566] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:24:18,345 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 29aede03
>2012-12-17 08:24:18,390 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [33a3b92a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:24:18,527 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-37) [783fb511] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:24:18,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5fc69a3c
>2012-12-17 08:24:18,580 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 34e029a0
>2012-12-17 08:24:18,599 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [783fb511] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:24:18,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, SpmStopVDSCommand, log id: 34e029a0
>2012-12-17 08:24:18,728 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, ResetIrsVDSCommand, log id: 5fc69a3c
>2012-12-17 08:24:18,740 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 70d3e5d3
>2012-12-17 08:24:18,740 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:24:18,740 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 70d3e5d3
>2012-12-17 08:24:18,810 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [381e4566] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:24:18,823 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [381e4566] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4530
>2012-12-17 08:24:18,825 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [381e4566] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4530, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 571879c4
>2012-12-17 08:24:18,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [381e4566] spmStart polling started: taskId = 9b34cb2b-e500-46d0-b812-724460d47989
>2012-12-17 08:24:19,869 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [381e4566] spmStart polling ended: taskId = 9b34cb2b-e500-46d0-b812-724460d47989 task status = finished
>2012-12-17 08:24:19,888 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [381e4566] spmStart polling ended, spm status: SPM
>2012-12-17 08:24:19,889 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [381e4566] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9b34cb2b-e500-46d0-b812-724460d47989), log id: 55121dec
>2012-12-17 08:24:19,907 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [381e4566] FINISH, HSMClearTaskVDSCommand, log id: 55121dec
>2012-12-17 08:24:19,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [381e4566] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@186a9c96, log id: 571879c4
>2012-12-17 08:24:19,917 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [381e4566] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:24:19,934 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:24:19,936 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 349124d9
>2012-12-17 08:24:19,954 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 349124d9
>2012-12-17 08:24:19,954 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 29aede03
>2012-12-17 08:24:19,955 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [381e4566] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:24:19,955 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 250a3d85
>2012-12-17 08:24:19,955 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:24:19,956 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2d9d8b27
>2012-12-17 08:24:19,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2d9d8b27
>2012-12-17 08:24:19,974 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 250a3d85
>2012-12-17 08:24:19,974 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [381e4566] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:24:27,865 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:24:28,877 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [2e10036] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:24:29,065 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [4579c6a9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:24:29,171 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [4579c6a9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 9fa5082
>2012-12-17 08:24:29,172 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4579c6a9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1158e3f6
>2012-12-17 08:24:29,192 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4579c6a9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:24:29,341 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4579c6a9] FINISH, SpmStopVDSCommand, log id: 1158e3f6
>2012-12-17 08:24:29,356 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [4579c6a9] FINISH, ResetIrsVDSCommand, log id: 9fa5082
>2012-12-17 08:24:29,366 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [4579c6a9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 36766928
>2012-12-17 08:24:29,366 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4579c6a9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:24:29,366 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [4579c6a9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 36766928
>2012-12-17 08:24:38,072 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [733ac637] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:24:39,417 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [3940cca7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:24:39,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [3940cca7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4531
>2012-12-17 08:24:39,428 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [3940cca7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4531, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 799c8a86
>2012-12-17 08:24:39,450 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [3940cca7] spmStart polling started: taskId = 8ca44242-b464-4775-bd44-a21c1db24ac2
>2012-12-17 08:24:40,471 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [3940cca7] spmStart polling ended: taskId = 8ca44242-b464-4775-bd44-a21c1db24ac2 task status = finished
>2012-12-17 08:24:40,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [3940cca7] spmStart polling ended, spm status: SPM
>2012-12-17 08:24:40,490 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [3940cca7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8ca44242-b464-4775-bd44-a21c1db24ac2), log id: 671eff28
>2012-12-17 08:24:40,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [3940cca7] FINISH, HSMClearTaskVDSCommand, log id: 671eff28
>2012-12-17 08:24:40,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [3940cca7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2401cffc, log id: 799c8a86
>2012-12-17 08:24:40,879 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [3940cca7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:24:41,393 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [5a89a26c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:24:41,393 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: cb0a668
>2012-12-17 08:24:41,718 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [3940cca7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:24:42,014 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [57b06faf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:24:42,110 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [57b06faf] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: a8177d2
>2012-12-17 08:24:42,115 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [57b06faf] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3e67524c
>2012-12-17 08:24:42,143 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [57b06faf] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:24:42,709 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [57b06faf] FINISH, SpmStopVDSCommand, log id: 3e67524c
>2012-12-17 08:24:42,724 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [57b06faf] FINISH, ResetIrsVDSCommand, log id: a8177d2
>2012-12-17 08:24:42,735 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [57b06faf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 432c6d65
>2012-12-17 08:24:42,735 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [57b06faf] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:24:42,735 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [57b06faf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 432c6d65
>2012-12-17 08:24:42,796 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:24:42,806 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4532
>2012-12-17 08:24:42,807 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4532, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 492a24ea
>2012-12-17 08:24:42,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] spmStart polling started: taskId = ff5e9ead-4bc7-4a9a-b2ea-dfddfa6240ad
>2012-12-17 08:24:43,855 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] spmStart polling ended: taskId = ff5e9ead-4bc7-4a9a-b2ea-dfddfa6240ad task status = finished
>2012-12-17 08:24:43,873 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] spmStart polling ended, spm status: SPM
>2012-12-17 08:24:43,875 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ff5e9ead-4bc7-4a9a-b2ea-dfddfa6240ad), log id: 1b921caa
>2012-12-17 08:24:43,894 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, HSMClearTaskVDSCommand, log id: 1b921caa
>2012-12-17 08:24:43,894 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1597eb0a, log id: 492a24ea
>2012-12-17 08:24:43,950 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:24:43,995 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:24:43,997 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 12f0af11
>2012-12-17 08:24:44,022 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 12f0af11
>2012-12-17 08:24:44,022 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: cb0a668
>2012-12-17 08:24:44,022 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [5a89a26c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:24:44,023 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 77758ecd
>2012-12-17 08:24:44,023 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:24:44,025 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4e683f16
>2012-12-17 08:24:44,043 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4e683f16
>2012-12-17 08:24:44,043 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 77758ecd
>2012-12-17 08:24:44,043 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [5a89a26c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:24:48,274 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [f50ed1e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:24:52,839 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7de89d45] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:24:53,023 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [65976fd0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:24:53,104 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 525215ea
>2012-12-17 08:24:53,106 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 644cc028
>2012-12-17 08:24:53,125 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:24:53,288 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, SpmStopVDSCommand, log id: 644cc028
>2012-12-17 08:24:53,301 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, ResetIrsVDSCommand, log id: 525215ea
>2012-12-17 08:24:53,313 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7f8fad0c
>2012-12-17 08:24:53,313 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [65976fd0] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:24:53,313 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7f8fad0c
>2012-12-17 08:24:58,523 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [6e64c0c8] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:25:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [2321a59a] Checking autorecoverable hosts
>2012-12-17 08:25:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [2321a59a] Autorecovering 0 hosts
>2012-12-17 08:25:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [2321a59a] Checking autorecoverable hosts done
>2012-12-17 08:25:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [2321a59a] Autorecovering storage domains is disabled, skipping
>2012-12-17 08:25:03,420 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [40098d4e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:25:03,444 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [40098d4e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4533
>2012-12-17 08:25:03,445 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [40098d4e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4533, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 629970ab
>2012-12-17 08:25:03,468 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [40098d4e] spmStart polling started: taskId = d2ca93c5-9cfa-43ff-8e77-48dff8d88a67
>2012-12-17 08:25:04,490 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [40098d4e] spmStart polling ended: taskId = d2ca93c5-9cfa-43ff-8e77-48dff8d88a67 task status = finished
>2012-12-17 08:25:04,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [40098d4e] spmStart polling ended, spm status: SPM
>2012-12-17 08:25:04,510 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [40098d4e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d2ca93c5-9cfa-43ff-8e77-48dff8d88a67), log id: 7efcc417
>2012-12-17 08:25:04,529 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [40098d4e] FINISH, HSMClearTaskVDSCommand, log id: 7efcc417
>2012-12-17 08:25:04,529 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [40098d4e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4ae0323, log id: 629970ab
>2012-12-17 08:25:04,552 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [40098d4e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:25:04,604 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:25:04,604 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 58639ce6
>2012-12-17 08:25:04,637 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [40098d4e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:25:04,761 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [311c1434] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:25:04,801 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [311c1434] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 73e1730
>2012-12-17 08:25:04,803 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [311c1434] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 54f6f5dc
>2012-12-17 08:25:04,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [311c1434] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:05,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [311c1434] FINISH, SpmStopVDSCommand, log id: 54f6f5dc
>2012-12-17 08:25:05,039 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [311c1434] FINISH, ResetIrsVDSCommand, log id: 73e1730
>2012-12-17 08:25:05,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [311c1434] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 62b258ec
>2012-12-17 08:25:05,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [311c1434] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:05,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [311c1434] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 62b258ec
>2012-12-17 08:25:05,091 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:25:05,101 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4534
>2012-12-17 08:25:05,103 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4534, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 54f1b836
>2012-12-17 08:25:05,125 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling started: taskId = 769ad798-7be4-4497-86a3-8594373d3c96
>2012-12-17 08:25:06,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended: taskId = 769ad798-7be4-4497-86a3-8594373d3c96 task status = finished
>2012-12-17 08:25:06,166 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended, spm status: SPM
>2012-12-17 08:25:06,168 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=769ad798-7be4-4497-86a3-8594373d3c96), log id: 1396e305
>2012-12-17 08:25:06,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMClearTaskVDSCommand, log id: 1396e305
>2012-12-17 08:25:06,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1712674f, log id: 54f1b836
>2012-12-17 08:25:06,199 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:25:06,223 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:06,227 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d49c9e6
>2012-12-17 08:25:06,247 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d49c9e6
>2012-12-17 08:25:06,247 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 58639ce6
>2012-12-17 08:25:06,247 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:25:06,247 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7665318
>2012-12-17 08:25:06,247 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:06,248 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7ee2225e
>2012-12-17 08:25:06,267 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7ee2225e
>2012-12-17 08:25:06,267 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7665318
>2012-12-17 08:25:06,267 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:25:08,769 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [64fd8799] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:25:15,140 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [6e8742f7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:25:15,827 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [5c1f293f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:25:15,969 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [5c1f293f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4d80c5b9
>2012-12-17 08:25:15,970 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [5c1f293f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2448544a
>2012-12-17 08:25:15,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [5c1f293f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:16,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [5c1f293f] FINISH, SpmStopVDSCommand, log id: 2448544a
>2012-12-17 08:25:16,171 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [5c1f293f] FINISH, ResetIrsVDSCommand, log id: 4d80c5b9
>2012-12-17 08:25:16,182 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [5c1f293f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 77f6e099
>2012-12-17 08:25:16,182 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [5c1f293f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:16,182 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [5c1f293f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 77f6e099
>2012-12-17 08:25:19,000 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [1c47e0ef] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:25:26,248 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [6fac8ea3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:25:26,262 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [6fac8ea3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4535
>2012-12-17 08:25:26,264 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [6fac8ea3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4535, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 70d55415
>2012-12-17 08:25:26,287 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [6fac8ea3] spmStart polling started: taskId = eb687825-b1d2-4833-be3f-d99c342899f3
>2012-12-17 08:25:27,310 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [6fac8ea3] spmStart polling ended: taskId = eb687825-b1d2-4833-be3f-d99c342899f3 task status = finished
>2012-12-17 08:25:27,328 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [6fac8ea3] spmStart polling ended, spm status: SPM
>2012-12-17 08:25:27,329 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [6fac8ea3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=eb687825-b1d2-4833-be3f-d99c342899f3), log id: 793709c5
>2012-12-17 08:25:27,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [6fac8ea3] FINISH, HSMClearTaskVDSCommand, log id: 793709c5
>2012-12-17 08:25:27,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [6fac8ea3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@c62a2af, log id: 70d55415
>2012-12-17 08:25:27,373 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [6fac8ea3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:25:27,410 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-73) [6bdbfd31] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:25:27,410 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5dafa7fc
>2012-12-17 08:25:27,434 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [6fac8ea3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:25:27,554 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [5a98cdcf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:25:27,622 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: ead29d9
>2012-12-17 08:25:27,624 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 273825e3
>2012-12-17 08:25:27,644 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:27,811 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, SpmStopVDSCommand, log id: 273825e3
>2012-12-17 08:25:27,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, ResetIrsVDSCommand, log id: ead29d9
>2012-12-17 08:25:27,839 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 688a5434
>2012-12-17 08:25:27,839 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [5a98cdcf] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:27,839 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 688a5434
>2012-12-17 08:25:27,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [6bdbfd31] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:25:27,928 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [6bdbfd31] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4536
>2012-12-17 08:25:27,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4536, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6fc1de2d
>2012-12-17 08:25:27,953 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] spmStart polling started: taskId = 69f0ce4d-a1f6-4abb-8516-5fb35fbebc10
>2012-12-17 08:25:28,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] spmStart polling ended: taskId = 69f0ce4d-a1f6-4abb-8516-5fb35fbebc10 task status = finished
>2012-12-17 08:25:28,994 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] spmStart polling ended, spm status: SPM
>2012-12-17 08:25:28,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=69f0ce4d-a1f6-4abb-8516-5fb35fbebc10), log id: 5a716c79
>2012-12-17 08:25:29,014 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, HSMClearTaskVDSCommand, log id: 5a716c79
>2012-12-17 08:25:29,014 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4c88aa52, log id: 6fc1de2d
>2012-12-17 08:25:29,031 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [6bdbfd31] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:25:29,049 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:29,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d5b238e
>2012-12-17 08:25:29,071 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d5b238e
>2012-12-17 08:25:29,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5dafa7fc
>2012-12-17 08:25:29,071 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-73) [6bdbfd31] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:25:29,072 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2dd6487c
>2012-12-17 08:25:29,072 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:29,073 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 16138d1f
>2012-12-17 08:25:29,093 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 16138d1f
>2012-12-17 08:25:29,093 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2dd6487c
>2012-12-17 08:25:29,093 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-73) [6bdbfd31] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:25:29,243 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [48fde2a8] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:25:37,987 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [1a57398] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:25:38,192 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [65c3c04c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:25:38,260 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [65c3c04c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3a7a58f6
>2012-12-17 08:25:38,261 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [65c3c04c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 69e8e155
>2012-12-17 08:25:38,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [65c3c04c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:39,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [65c3c04c] FINISH, SpmStopVDSCommand, log id: 69e8e155
>2012-12-17 08:25:39,049 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [65c3c04c] FINISH, ResetIrsVDSCommand, log id: 3a7a58f6
>2012-12-17 08:25:39,060 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [65c3c04c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49f73630
>2012-12-17 08:25:39,060 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [65c3c04c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:39,060 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [65c3c04c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49f73630
>2012-12-17 08:25:39,455 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [53e8b92c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:25:49,111 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [d425472] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:25:49,143 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [d425472] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4537
>2012-12-17 08:25:49,144 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [d425472] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4537, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5157b0
>2012-12-17 08:25:49,168 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [d425472] spmStart polling started: taskId = 19e2515e-01f6-41fb-ad2f-80e2266c827b
>2012-12-17 08:25:50,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [d425472] spmStart polling ended: taskId = 19e2515e-01f6-41fb-ad2f-80e2266c827b task status = finished
>2012-12-17 08:25:50,208 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [d425472] spmStart polling ended, spm status: SPM
>2012-12-17 08:25:50,209 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [d425472] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=19e2515e-01f6-41fb-ad2f-80e2266c827b), log id: 60e593e7
>2012-12-17 08:25:50,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [d425472] FINISH, HSMClearTaskVDSCommand, log id: 60e593e7
>2012-12-17 08:25:50,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [d425472] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@515464, log id: 5157b0
>2012-12-17 08:25:50,812 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [d425472] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:25:51,438 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [76a14bbb] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:25:51,438 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1bedef9e
>2012-12-17 08:25:51,460 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [d425472] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:25:51,762 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [3e6ad503] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:25:51,774 INFO  [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-57) [55210d2] DbUserCacheManager::refreshAllUserData() - entered
>2012-12-17 08:25:51,866 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [3e6ad503] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3d232026
>2012-12-17 08:25:51,867 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [3e6ad503] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 23186edd
>2012-12-17 08:25:51,887 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [3e6ad503] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:52,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [3e6ad503] FINISH, SpmStopVDSCommand, log id: 23186edd
>2012-12-17 08:25:52,668 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [3e6ad503] FINISH, ResetIrsVDSCommand, log id: 3d232026
>2012-12-17 08:25:52,680 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [3e6ad503] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 14287446
>2012-12-17 08:25:52,680 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3e6ad503] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:52,680 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [3e6ad503] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 14287446
>2012-12-17 08:25:52,740 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [76a14bbb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:25:52,756 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [76a14bbb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4538
>2012-12-17 08:25:52,758 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4538, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 20c3733a
>2012-12-17 08:25:52,782 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] spmStart polling started: taskId = 5859914b-f5c7-4308-9ace-246152dc68a9
>2012-12-17 08:25:53,803 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] spmStart polling ended: taskId = 5859914b-f5c7-4308-9ace-246152dc68a9 task status = finished
>2012-12-17 08:25:53,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] spmStart polling ended, spm status: SPM
>2012-12-17 08:25:53,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5859914b-f5c7-4308-9ace-246152dc68a9), log id: 4838c30e
>2012-12-17 08:25:53,841 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, HSMClearTaskVDSCommand, log id: 4838c30e
>2012-12-17 08:25:53,841 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@68df12d, log id: 20c3733a
>2012-12-17 08:25:53,874 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [76a14bbb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:25:53,907 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:53,909 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4622bfe0
>2012-12-17 08:25:53,928 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4622bfe0
>2012-12-17 08:25:53,928 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1bedef9e
>2012-12-17 08:25:53,928 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [76a14bbb] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:25:53,929 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 54a91c40
>2012-12-17 08:25:53,929 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:25:53,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 623618cf
>2012-12-17 08:25:53,949 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 623618cf
>2012-12-17 08:25:53,949 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 54a91c40
>2012-12-17 08:25:53,949 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [76a14bbb] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:25:59,989 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:26:02,790 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [5c1f293f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:26:02,940 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [67020b95] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:26:03,020 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2bb0db57
>2012-12-17 08:26:03,022 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6e8e78ea
>2012-12-17 08:26:03,045 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [67020b95] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:26:03,190 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, SpmStopVDSCommand, log id: 6e8e78ea
>2012-12-17 08:26:03,256 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, ResetIrsVDSCommand, log id: 2bb0db57
>2012-12-17 08:26:03,268 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d2a1b94
>2012-12-17 08:26:03,268 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:26:03,268 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d2a1b94
>2012-12-17 08:26:10,189 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:26:13,343 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1852110b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:26:13,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1852110b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4539
>2012-12-17 08:26:13,369 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [1852110b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4539, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5ae92d37
>2012-12-17 08:26:13,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [1852110b] spmStart polling started: taskId = b926e5a5-cc04-4425-82e8-1b783d9deef3
>2012-12-17 08:26:14,416 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [1852110b] spmStart polling ended: taskId = b926e5a5-cc04-4425-82e8-1b783d9deef3 task status = finished
>2012-12-17 08:26:14,434 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [1852110b] spmStart polling ended, spm status: SPM
>2012-12-17 08:26:14,435 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [1852110b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b926e5a5-cc04-4425-82e8-1b783d9deef3), log id: bdb6713
>2012-12-17 08:26:14,454 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [1852110b] FINISH, HSMClearTaskVDSCommand, log id: bdb6713
>2012-12-17 08:26:14,454 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [1852110b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@714c1679, log id: 5ae92d37
>2012-12-17 08:26:14,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1852110b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:26:14,603 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [5a98cdcf] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:26:14,603 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5b233a27
>2012-12-17 08:26:14,669 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1852110b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:26:14,849 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [5863fdac] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:26:14,925 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 32e6638a
>2012-12-17 08:26:14,927 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6d226762
>2012-12-17 08:26:14,947 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:26:15,116 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] FINISH, SpmStopVDSCommand, log id: 6d226762
>2012-12-17 08:26:15,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] FINISH, ResetIrsVDSCommand, log id: 32e6638a
>2012-12-17 08:26:15,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b0cf9a5
>2012-12-17 08:26:15,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [5863fdac] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:26:15,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b0cf9a5
>2012-12-17 08:26:15,215 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [5a98cdcf] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:26:16,202 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [5a98cdcf] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4540
>2012-12-17 08:26:16,204 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4540, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 76e139c1
>2012-12-17 08:26:16,227 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] spmStart polling started: taskId = 57201b7e-e6d7-4554-ba80-8c6cf330f9dd
>2012-12-17 08:26:17,277 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] spmStart polling ended: taskId = 57201b7e-e6d7-4554-ba80-8c6cf330f9dd task status = finished
>2012-12-17 08:26:17,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] spmStart polling ended, spm status: SPM
>2012-12-17 08:26:17,299 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=57201b7e-e6d7-4554-ba80-8c6cf330f9dd), log id: a105d28
>2012-12-17 08:26:17,318 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, HSMClearTaskVDSCommand, log id: a105d28
>2012-12-17 08:26:17,318 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3853f241, log id: 76e139c1
>2012-12-17 08:26:17,338 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [5a98cdcf] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:26:17,359 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:26:17,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 771c9e3d
>2012-12-17 08:26:17,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 771c9e3d
>2012-12-17 08:26:17,381 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5b233a27
>2012-12-17 08:26:17,381 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [5a98cdcf] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:26:17,381 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4473b7e6
>2012-12-17 08:26:17,381 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:26:17,383 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: b628257
>2012-12-17 08:26:17,401 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: b628257
>2012-12-17 08:26:17,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4473b7e6
>2012-12-17 08:26:17,402 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [5a98cdcf] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:26:20,788 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [717cca29] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:26:25,379 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [65c3c04c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:26:26,216 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [152503f5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:26:26,311 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [152503f5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 548bed55
>2012-12-17 08:26:26,312 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [152503f5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 23ecef64
>2012-12-17 08:26:26,332 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [152503f5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:26:26,479 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [152503f5] FINISH, SpmStopVDSCommand, log id: 23ecef64
>2012-12-17 08:26:26,502 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [152503f5] FINISH, ResetIrsVDSCommand, log id: 548bed55
>2012-12-17 08:26:26,512 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [152503f5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 47c9aeb2
>2012-12-17 08:26:26,513 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [152503f5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:26:26,513 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [152503f5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 47c9aeb2
>2012-12-17 08:26:31,620 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:26:36,560 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6995ddd2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:26:36,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6995ddd2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4541
>2012-12-17 08:26:36,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6995ddd2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4541, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 514908fd
>2012-12-17 08:26:36,627 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6995ddd2] spmStart polling started: taskId = dbecc6ac-9776-4181-83ce-7673e9e7116e
>2012-12-17 08:26:37,650 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6995ddd2] spmStart polling ended: taskId = dbecc6ac-9776-4181-83ce-7673e9e7116e task status = finished
>2012-12-17 08:26:37,668 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6995ddd2] spmStart polling ended, spm status: SPM
>2012-12-17 08:26:37,669 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [6995ddd2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dbecc6ac-9776-4181-83ce-7673e9e7116e), log id: 73b02d14
>2012-12-17 08:26:37,699 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [6995ddd2] FINISH, HSMClearTaskVDSCommand, log id: 73b02d14
>2012-12-17 08:26:37,699 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6995ddd2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6aa261a6, log id: 514908fd
>2012-12-17 08:26:37,717 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6995ddd2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:26:37,764 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:26:37,764 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7df0bb6b
>2012-12-17 08:26:37,784 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6995ddd2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:26:37,998 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [1eda0292] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:26:38,207 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [1eda0292] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6a3a2fc6
>2012-12-17 08:26:38,209 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [1eda0292] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 680a596e
>2012-12-17 08:26:38,229 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [1eda0292] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:26:38,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [1eda0292] FINISH, SpmStopVDSCommand, log id: 680a596e
>2012-12-17 08:26:38,454 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [1eda0292] FINISH, ResetIrsVDSCommand, log id: 6a3a2fc6
>2012-12-17 08:26:38,465 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [1eda0292] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4771ecb5
>2012-12-17 08:26:38,465 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1eda0292] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:26:38,465 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [1eda0292] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4771ecb5
>2012-12-17 08:26:38,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:26:38,555 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4542
>2012-12-17 08:26:38,556 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4542, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40a93a5d
>2012-12-17 08:26:38,578 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling started: taskId = a092db07-2a7c-47ae-a134-137a45cd4354
>2012-12-17 08:26:39,600 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling ended: taskId = a092db07-2a7c-47ae-a134-137a45cd4354 task status = finished
>2012-12-17 08:26:39,619 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling ended, spm status: SPM
>2012-12-17 08:26:39,620 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a092db07-2a7c-47ae-a134-137a45cd4354), log id: 6efac562
>2012-12-17 08:26:39,638 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMClearTaskVDSCommand, log id: 6efac562
>2012-12-17 08:26:39,638 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@11b4ddda, log id: 40a93a5d
>2012-12-17 08:26:39,681 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:26:39,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:26:39,698 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 21eb2a3d
>2012-12-17 08:26:39,733 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 21eb2a3d
>2012-12-17 08:26:39,733 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7df0bb6b
>2012-12-17 08:26:39,733 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:26:39,734 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: aa62346
>2012-12-17 08:26:39,734 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:26:39,735 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 49ce1464
>2012-12-17 08:26:39,754 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 49ce1464
>2012-12-17 08:26:39,754 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: aa62346
>2012-12-17 08:26:39,754 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:26:41,814 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [76a14bbb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:26:48,597 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:26:48,727 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-12) [664081c5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:26:48,769 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [664081c5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 71c8c33c
>2012-12-17 08:26:48,771 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [664081c5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 176ba17a
>2012-12-17 08:26:48,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [664081c5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:26:49,425 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [664081c5] FINISH, SpmStopVDSCommand, log id: 176ba17a
>2012-12-17 08:26:49,440 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [664081c5] FINISH, ResetIrsVDSCommand, log id: 71c8c33c
>2012-12-17 08:26:49,451 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [664081c5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4150ab26
>2012-12-17 08:26:49,451 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [664081c5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:26:49,451 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [664081c5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4150ab26
>2012-12-17 08:26:52,037 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [376f3052] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:26:59,504 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3ae71176] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:26:59,518 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3ae71176] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4543
>2012-12-17 08:26:59,519 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4543, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7219d0c2
>2012-12-17 08:26:59,541 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] spmStart polling started: taskId = 5b6b6902-4e29-4047-b177-c66353d712d3
>2012-12-17 08:27:00,564 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] spmStart polling ended: taskId = 5b6b6902-4e29-4047-b177-c66353d712d3 task status = finished
>2012-12-17 08:27:00,583 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] spmStart polling ended, spm status: SPM
>2012-12-17 08:27:00,584 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5b6b6902-4e29-4047-b177-c66353d712d3), log id: 56f658c2
>2012-12-17 08:27:00,604 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] FINISH, HSMClearTaskVDSCommand, log id: 56f658c2
>2012-12-17 08:27:00,604 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@473d5528, log id: 7219d0c2
>2012-12-17 08:27:00,783 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3ae71176] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:27:01,088 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:27:01,089 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 38729bd7
>2012-12-17 08:27:01,161 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3ae71176] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:27:01,391 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [50f32e72] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:27:01,432 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 534fec55
>2012-12-17 08:27:01,434 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: decf640
>2012-12-17 08:27:01,454 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:01,573 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, SpmStopVDSCommand, log id: decf640
>2012-12-17 08:27:01,595 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, ResetIrsVDSCommand, log id: 534fec55
>2012-12-17 08:27:01,605 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ef64bb9
>2012-12-17 08:27:01,606 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:01,606 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ef64bb9
>2012-12-17 08:27:01,646 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:27:01,674 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4544
>2012-12-17 08:27:01,675 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4544, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67be27d8
>2012-12-17 08:27:01,698 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling started: taskId = 9267a991-579a-4158-9671-fe922c935fe5
>2012-12-17 08:27:02,719 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended: taskId = 9267a991-579a-4158-9671-fe922c935fe5 task status = finished
>2012-12-17 08:27:02,737 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended, spm status: SPM
>2012-12-17 08:27:02,738 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9267a991-579a-4158-9671-fe922c935fe5), log id: 63b5c46f
>2012-12-17 08:27:02,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMClearTaskVDSCommand, log id: 63b5c46f
>2012-12-17 08:27:02,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3273868d, log id: 67be27d8
>2012-12-17 08:27:02,767 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:27:02,782 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:02,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4ccd6597
>2012-12-17 08:27:02,804 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4ccd6597
>2012-12-17 08:27:02,804 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 38729bd7
>2012-12-17 08:27:02,804 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:27:02,804 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3cb5109d
>2012-12-17 08:27:02,804 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:02,805 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3a93888a
>2012-12-17 08:27:02,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3a93888a
>2012-12-17 08:27:02,823 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3cb5109d
>2012-12-17 08:27:02,824 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:27:11,685 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [3116bd71] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:27:11,819 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [69b76134] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:27:11,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [69b76134] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 616b9f37
>2012-12-17 08:27:11,889 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [69b76134] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1647fb5b
>2012-12-17 08:27:11,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [69b76134] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:12,079 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [69b76134] FINISH, SpmStopVDSCommand, log id: 1647fb5b
>2012-12-17 08:27:12,100 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [69b76134] FINISH, ResetIrsVDSCommand, log id: 616b9f37
>2012-12-17 08:27:12,111 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [69b76134] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1585de32
>2012-12-17 08:27:12,111 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [69b76134] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:12,111 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [69b76134] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1585de32
>2012-12-17 08:27:12,447 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:27:22,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [2321a59a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:27:22,196 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [2321a59a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4545
>2012-12-17 08:27:22,198 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [2321a59a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4545, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 490c48ea
>2012-12-17 08:27:22,220 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [2321a59a] spmStart polling started: taskId = 9ac5a3b8-2000-46af-92dd-0ec34042d29d
>2012-12-17 08:27:23,241 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [2321a59a] spmStart polling ended: taskId = 9ac5a3b8-2000-46af-92dd-0ec34042d29d task status = finished
>2012-12-17 08:27:23,260 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [2321a59a] spmStart polling ended, spm status: SPM
>2012-12-17 08:27:23,261 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [2321a59a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9ac5a3b8-2000-46af-92dd-0ec34042d29d), log id: 69ce8866
>2012-12-17 08:27:23,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [2321a59a] FINISH, HSMClearTaskVDSCommand, log id: 69ce8866
>2012-12-17 08:27:23,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [2321a59a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@765d16ce, log id: 490c48ea
>2012-12-17 08:27:23,298 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [2321a59a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:27:23,349 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [3b972285] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:27:23,349 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2a96fba3
>2012-12-17 08:27:23,365 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [2321a59a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:27:23,642 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-100) [5b075498] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:27:23,782 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [5b075498] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4e0d18fa
>2012-12-17 08:27:23,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [5b075498] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7eda55dc
>2012-12-17 08:27:23,807 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [5b075498] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:23,932 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [5b075498] FINISH, SpmStopVDSCommand, log id: 7eda55dc
>2012-12-17 08:27:23,941 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [5b075498] FINISH, ResetIrsVDSCommand, log id: 4e0d18fa
>2012-12-17 08:27:23,952 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [5b075498] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4b85a3a7
>2012-12-17 08:27:23,952 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [5b075498] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:23,952 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [5b075498] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4b85a3a7
>2012-12-17 08:27:24,014 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:27:24,042 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4546
>2012-12-17 08:27:24,043 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4546, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7b722591
>2012-12-17 08:27:24,065 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] spmStart polling started: taskId = 8a1cc935-6bc1-40e6-98d3-c05ecf197826
>2012-12-17 08:27:25,088 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] spmStart polling ended: taskId = 8a1cc935-6bc1-40e6-98d3-c05ecf197826 task status = finished
>2012-12-17 08:27:25,106 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] spmStart polling ended, spm status: SPM
>2012-12-17 08:27:25,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8a1cc935-6bc1-40e6-98d3-c05ecf197826), log id: 1b23d50c
>2012-12-17 08:27:25,127 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, HSMClearTaskVDSCommand, log id: 1b23d50c
>2012-12-17 08:27:25,127 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4d7814bf, log id: 7b722591
>2012-12-17 08:27:25,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:27:25,162 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:25,166 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6f93c5ce
>2012-12-17 08:27:25,185 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f93c5ce
>2012-12-17 08:27:25,185 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2a96fba3
>2012-12-17 08:27:25,185 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [3b972285] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:27:25,185 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4e9f909c
>2012-12-17 08:27:25,185 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:25,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 52acef5f
>2012-12-17 08:27:25,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 52acef5f
>2012-12-17 08:27:25,207 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4e9f909c
>2012-12-17 08:27:25,207 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [3b972285] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:27:32,831 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [36fcbeec] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:27:34,059 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [7b78f9c5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:27:34,200 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-29) [4adae006] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:27:34,250 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [4adae006] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 680e741
>2012-12-17 08:27:34,252 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [4adae006] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 62949086
>2012-12-17 08:27:34,272 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [4adae006] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:34,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [4adae006] FINISH, SpmStopVDSCommand, log id: 62949086
>2012-12-17 08:27:34,443 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [4adae006] FINISH, ResetIrsVDSCommand, log id: 680e741
>2012-12-17 08:27:34,454 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [4adae006] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44fba90c
>2012-12-17 08:27:34,454 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [4adae006] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:34,454 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [4adae006] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44fba90c
>2012-12-17 08:27:43,032 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:27:44,498 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [13794c1c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:27:44,514 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [13794c1c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4547
>2012-12-17 08:27:44,516 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [13794c1c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4547, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2fb4195d
>2012-12-17 08:27:44,539 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [13794c1c] spmStart polling started: taskId = f7dc1354-ad82-4ee7-9f16-e0fe47cfcd3f
>2012-12-17 08:27:45,561 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [13794c1c] spmStart polling ended: taskId = f7dc1354-ad82-4ee7-9f16-e0fe47cfcd3f task status = finished
>2012-12-17 08:27:45,585 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [13794c1c] spmStart polling ended, spm status: SPM
>2012-12-17 08:27:45,586 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [13794c1c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f7dc1354-ad82-4ee7-9f16-e0fe47cfcd3f), log id: 3469e4e6
>2012-12-17 08:27:45,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [13794c1c] FINISH, HSMClearTaskVDSCommand, log id: 3469e4e6
>2012-12-17 08:27:45,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [13794c1c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1432d2ec, log id: 2fb4195d
>2012-12-17 08:27:45,999 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [13794c1c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:27:46,035 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [406132e6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:27:46,036 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1a105a08
>2012-12-17 08:27:46,113 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [13794c1c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:27:46,276 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-51) [5b5bb621] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:27:46,322 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5698c284
>2012-12-17 08:27:46,324 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f0a6cd9
>2012-12-17 08:27:46,345 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:46,494 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] FINISH, SpmStopVDSCommand, log id: 4f0a6cd9
>2012-12-17 08:27:46,503 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] FINISH, ResetIrsVDSCommand, log id: 5698c284
>2012-12-17 08:27:46,515 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 21e2bf06
>2012-12-17 08:27:46,515 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b5bb621] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:46,515 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 21e2bf06
>2012-12-17 08:27:46,563 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:27:46,577 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4548
>2012-12-17 08:27:46,578 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4548, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 125b03d9
>2012-12-17 08:27:46,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] spmStart polling started: taskId = 7ba4e369-cd91-4f1a-aa29-7c36750e15da
>2012-12-17 08:27:47,623 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] spmStart polling ended: taskId = 7ba4e369-cd91-4f1a-aa29-7c36750e15da task status = finished
>2012-12-17 08:27:47,646 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] spmStart polling ended, spm status: SPM
>2012-12-17 08:27:47,647 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7ba4e369-cd91-4f1a-aa29-7c36750e15da), log id: 79ca7c7
>2012-12-17 08:27:47,666 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, HSMClearTaskVDSCommand, log id: 79ca7c7
>2012-12-17 08:27:47,666 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3dcafdb8, log id: 125b03d9
>2012-12-17 08:27:47,689 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:27:47,701 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:47,702 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 135e7da8
>2012-12-17 08:27:47,724 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 135e7da8
>2012-12-17 08:27:47,724 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1a105a08
>2012-12-17 08:27:47,724 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [406132e6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:27:47,724 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 316895c2
>2012-12-17 08:27:47,724 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:47,725 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6e99b9ac
>2012-12-17 08:27:47,744 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6e99b9ac
>2012-12-17 08:27:47,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 316895c2
>2012-12-17 08:27:47,744 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [406132e6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:27:53,217 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:27:56,610 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [24da4704] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:27:56,771 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [a209fdc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:27:56,840 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 473fa876
>2012-12-17 08:27:56,842 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 498dd5cd
>2012-12-17 08:27:56,862 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:57,016 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] FINISH, SpmStopVDSCommand, log id: 498dd5cd
>2012-12-17 08:27:57,029 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] FINISH, ResetIrsVDSCommand, log id: 473fa876
>2012-12-17 08:27:57,040 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1e1fa20d
>2012-12-17 08:27:57,040 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [a209fdc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:27:57,040 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1e1fa20d
>2012-12-17 08:28:03,429 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [17cf9677] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:28:07,113 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [43662b97] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:28:07,129 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [43662b97] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4549
>2012-12-17 08:28:07,141 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [43662b97] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4549, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 12aa38ff
>2012-12-17 08:28:07,163 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [43662b97] spmStart polling started: taskId = f9d24c1e-3597-4134-8a1c-d3055e248267
>2012-12-17 08:28:08,184 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [43662b97] spmStart polling ended: taskId = f9d24c1e-3597-4134-8a1c-d3055e248267 task status = finished
>2012-12-17 08:28:08,202 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [43662b97] spmStart polling ended, spm status: SPM
>2012-12-17 08:28:08,203 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [43662b97] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f9d24c1e-3597-4134-8a1c-d3055e248267), log id: 7b40c319
>2012-12-17 08:28:08,222 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [43662b97] FINISH, HSMClearTaskVDSCommand, log id: 7b40c319
>2012-12-17 08:28:08,222 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [43662b97] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@58c8dd86, log id: 12aa38ff
>2012-12-17 08:28:08,259 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [43662b97] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:28:08,304 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [4f7e897c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:28:08,305 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 78853209
>2012-12-17 08:28:08,325 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [43662b97] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:28:08,479 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [1a2cf793] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:28:08,560 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [1a2cf793] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1894fd64
>2012-12-17 08:28:08,561 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1a2cf793] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 79e7068d
>2012-12-17 08:28:08,582 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1a2cf793] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:09,221 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1a2cf793] FINISH, SpmStopVDSCommand, log id: 79e7068d
>2012-12-17 08:28:09,260 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [1a2cf793] FINISH, ResetIrsVDSCommand, log id: 1894fd64
>2012-12-17 08:28:09,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [1a2cf793] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1339bb6a
>2012-12-17 08:28:09,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1a2cf793] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:09,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [1a2cf793] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1339bb6a
>2012-12-17 08:28:09,331 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [4f7e897c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:28:09,338 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [4f7e897c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4550
>2012-12-17 08:28:09,339 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4550, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2bdad4b2
>2012-12-17 08:28:09,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] spmStart polling started: taskId = da659046-1dcf-41d9-a9e6-3eb5a3f1e443
>2012-12-17 08:28:10,383 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] spmStart polling ended: taskId = da659046-1dcf-41d9-a9e6-3eb5a3f1e443 task status = finished
>2012-12-17 08:28:10,401 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] spmStart polling ended, spm status: SPM
>2012-12-17 08:28:10,403 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=da659046-1dcf-41d9-a9e6-3eb5a3f1e443), log id: 6f3fbfed
>2012-12-17 08:28:10,421 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] FINISH, HSMClearTaskVDSCommand, log id: 6f3fbfed
>2012-12-17 08:28:10,421 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@425199aa, log id: 2bdad4b2
>2012-12-17 08:28:10,897 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [4f7e897c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:28:10,960 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:10,962 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7e0f8ee5
>2012-12-17 08:28:10,981 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7e0f8ee5
>2012-12-17 08:28:10,981 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 78853209
>2012-12-17 08:28:10,981 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [4f7e897c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:28:10,981 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2f6e9d0b
>2012-12-17 08:28:10,981 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:10,982 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2ccacb34
>2012-12-17 08:28:11,000 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2ccacb34
>2012-12-17 08:28:11,000 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2f6e9d0b
>2012-12-17 08:28:11,000 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [4f7e897c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:28:13,686 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [311c1434] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:28:19,367 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [64fd8799] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:28:19,464 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-28) [5571991a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:28:19,518 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [5571991a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1561d894
>2012-12-17 08:28:19,519 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [5571991a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f05c822
>2012-12-17 08:28:19,539 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [5571991a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:19,763 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [5571991a] FINISH, SpmStopVDSCommand, log id: 4f05c822
>2012-12-17 08:28:19,778 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [5571991a] FINISH, ResetIrsVDSCommand, log id: 1561d894
>2012-12-17 08:28:19,790 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [5571991a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7cd73ff0
>2012-12-17 08:28:19,790 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [5571991a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:19,790 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [5571991a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7cd73ff0
>2012-12-17 08:28:23,892 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1a53ba46] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:28:29,857 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [61c8d0e2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:28:29,878 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [61c8d0e2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4551
>2012-12-17 08:28:29,880 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [61c8d0e2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4551, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: e97d4b5
>2012-12-17 08:28:29,901 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [61c8d0e2] spmStart polling started: taskId = a0368716-d7f3-48e4-82a5-acd5be6ec3a1
>2012-12-17 08:28:30,923 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [61c8d0e2] spmStart polling ended: taskId = a0368716-d7f3-48e4-82a5-acd5be6ec3a1 task status = finished
>2012-12-17 08:28:30,946 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [61c8d0e2] spmStart polling ended, spm status: SPM
>2012-12-17 08:28:30,947 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [61c8d0e2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a0368716-d7f3-48e4-82a5-acd5be6ec3a1), log id: fd29875
>2012-12-17 08:28:30,967 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [61c8d0e2] FINISH, HSMClearTaskVDSCommand, log id: fd29875
>2012-12-17 08:28:30,967 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [61c8d0e2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4b173170, log id: e97d4b5
>2012-12-17 08:28:30,999 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [61c8d0e2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:28:31,055 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:28:31,055 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 58a0a6c0
>2012-12-17 08:28:31,066 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [61c8d0e2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:28:31,286 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [550b34b1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:28:31,364 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [550b34b1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6e95a248
>2012-12-17 08:28:31,366 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [550b34b1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 16febfc7
>2012-12-17 08:28:31,387 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [550b34b1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:32,184 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [550b34b1] FINISH, SpmStopVDSCommand, log id: 16febfc7
>2012-12-17 08:28:32,203 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [550b34b1] FINISH, ResetIrsVDSCommand, log id: 6e95a248
>2012-12-17 08:28:32,214 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [550b34b1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c5cde92
>2012-12-17 08:28:32,214 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [550b34b1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:32,214 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [550b34b1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c5cde92
>2012-12-17 08:28:32,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:28:32,289 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4552
>2012-12-17 08:28:32,291 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4552, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 141d1138
>2012-12-17 08:28:32,313 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling started: taskId = 354f971f-5700-4629-9aa7-e4c6b67757db
>2012-12-17 08:28:33,335 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended: taskId = 354f971f-5700-4629-9aa7-e4c6b67757db task status = finished
>2012-12-17 08:28:33,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended, spm status: SPM
>2012-12-17 08:28:33,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=354f971f-5700-4629-9aa7-e4c6b67757db), log id: f982ff8
>2012-12-17 08:28:33,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMClearTaskVDSCommand, log id: f982ff8
>2012-12-17 08:28:33,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@50754ac7, log id: 141d1138
>2012-12-17 08:28:33,393 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:28:33,436 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:33,440 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1ebbb714
>2012-12-17 08:28:33,459 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ebbb714
>2012-12-17 08:28:33,459 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 58a0a6c0
>2012-12-17 08:28:33,459 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:28:33,459 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 35f091cf
>2012-12-17 08:28:33,459 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:33,461 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1ac5b316
>2012-12-17 08:28:33,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ac5b316
>2012-12-17 08:28:33,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 35f091cf
>2012-12-17 08:28:33,480 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:28:34,178 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [733ac637] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:28:42,330 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [723e6c9b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:28:42,483 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [28319bba] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:28:42,574 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3464f905
>2012-12-17 08:28:42,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5887b2aa
>2012-12-17 08:28:42,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [28319bba] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:43,434 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SpmStopVDSCommand, log id: 5887b2aa
>2012-12-17 08:28:43,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, ResetIrsVDSCommand, log id: 3464f905
>2012-12-17 08:28:43,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24c6c50d
>2012-12-17 08:28:43,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:43,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24c6c50d
>2012-12-17 08:28:44,373 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:28:53,535 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5f87b09d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:28:53,552 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5f87b09d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4553
>2012-12-17 08:28:53,553 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f87b09d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4553, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3ca0bf09
>2012-12-17 08:28:53,575 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f87b09d] spmStart polling started: taskId = f9a60457-7b64-4825-827c-c3ab45d13c0b
>2012-12-17 08:28:54,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f87b09d] spmStart polling ended: taskId = f9a60457-7b64-4825-827c-c3ab45d13c0b task status = finished
>2012-12-17 08:28:54,613 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f87b09d] spmStart polling ended, spm status: SPM
>2012-12-17 08:28:54,614 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [5f87b09d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f9a60457-7b64-4825-827c-c3ab45d13c0b), log id: 3b90741d
>2012-12-17 08:28:54,633 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [5f87b09d] FINISH, HSMClearTaskVDSCommand, log id: 3b90741d
>2012-12-17 08:28:54,633 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f87b09d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@8b73a78, log id: 3ca0bf09
>2012-12-17 08:28:54,672 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5f87b09d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:28:54,701 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [2b22186] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:28:54,701 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 77a18334
>2012-12-17 08:28:54,729 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5f87b09d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:28:54,824 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [780c369f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:28:54,889 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [780c369f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1268ec8f
>2012-12-17 08:28:54,890 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [780c369f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6ecc85d9
>2012-12-17 08:28:54,910 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [780c369f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:55,076 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [780c369f] FINISH, SpmStopVDSCommand, log id: 6ecc85d9
>2012-12-17 08:28:55,116 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [780c369f] FINISH, ResetIrsVDSCommand, log id: 1268ec8f
>2012-12-17 08:28:55,127 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [780c369f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 473d11d4
>2012-12-17 08:28:55,127 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [780c369f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:55,127 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [780c369f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 473d11d4
>2012-12-17 08:28:55,170 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [2b22186] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:28:55,183 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [2b22186] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4554
>2012-12-17 08:28:55,184 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [2b22186] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4554, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6e64bff1
>2012-12-17 08:28:55,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [2b22186] spmStart polling started: taskId = a6036110-6bbb-4162-9520-2227b694d8bb
>2012-12-17 08:28:56,229 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [2b22186] spmStart polling ended: taskId = a6036110-6bbb-4162-9520-2227b694d8bb task status = finished
>2012-12-17 08:28:56,248 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [2b22186] spmStart polling ended, spm status: SPM
>2012-12-17 08:28:56,249 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [2b22186] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a6036110-6bbb-4162-9520-2227b694d8bb), log id: 76fbcddb
>2012-12-17 08:28:56,268 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [2b22186] FINISH, HSMClearTaskVDSCommand, log id: 76fbcddb
>2012-12-17 08:28:56,268 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [2b22186] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@d8821e4, log id: 6e64bff1
>2012-12-17 08:28:56,279 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [2b22186] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:28:56,302 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:56,304 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 13fec599
>2012-12-17 08:28:56,323 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 13fec599
>2012-12-17 08:28:56,323 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 77a18334
>2012-12-17 08:28:56,323 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [2b22186] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:28:56,323 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4c23fee3
>2012-12-17 08:28:56,323 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:28:56,325 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 12a62ee
>2012-12-17 08:28:56,343 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 12a62ee
>2012-12-17 08:28:56,343 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4c23fee3
>2012-12-17 08:28:56,343 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [2b22186] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:29:04,777 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [50c114a6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:29:05,217 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [76a14bbb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:29:06,616 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [1981e345] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:29:06,737 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 121a7acd
>2012-12-17 08:29:06,738 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2074534
>2012-12-17 08:29:06,759 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [1981e345] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:29:06,944 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, SpmStopVDSCommand, log id: 2074534
>2012-12-17 08:29:06,957 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, ResetIrsVDSCommand, log id: 121a7acd
>2012-12-17 08:29:06,968 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: ad93aa1
>2012-12-17 08:29:06,968 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1981e345] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:29:06,968 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, MarkPoolInReconstructModeVDSCommand, log id: ad93aa1
>2012-12-17 08:29:14,962 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:29:17,017 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [6462735f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:29:17,030 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [6462735f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4555
>2012-12-17 08:29:17,032 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [6462735f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4555, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 55af6880
>2012-12-17 08:29:17,053 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [6462735f] spmStart polling started: taskId = ae0410fe-65fd-47e8-a52a-2ff86a2e7e13
>2012-12-17 08:29:18,076 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [6462735f] spmStart polling ended: taskId = ae0410fe-65fd-47e8-a52a-2ff86a2e7e13 task status = finished
>2012-12-17 08:29:18,094 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [6462735f] spmStart polling ended, spm status: SPM
>2012-12-17 08:29:18,096 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [6462735f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ae0410fe-65fd-47e8-a52a-2ff86a2e7e13), log id: 6ef9b666
>2012-12-17 08:29:18,114 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [6462735f] FINISH, HSMClearTaskVDSCommand, log id: 6ef9b666
>2012-12-17 08:29:18,114 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [6462735f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4c361b75, log id: 55af6880
>2012-12-17 08:29:18,136 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [6462735f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:29:18,296 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [54cb0ab7] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:29:18,296 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 114c1056
>2012-12-17 08:29:18,308 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [6462735f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:29:18,506 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [65048227] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:29:18,589 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 20390a05
>2012-12-17 08:29:18,591 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: e4449d9
>2012-12-17 08:29:18,611 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [65048227] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:29:18,745 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, SpmStopVDSCommand, log id: e4449d9
>2012-12-17 08:29:18,771 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, ResetIrsVDSCommand, log id: 20390a05
>2012-12-17 08:29:18,783 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 556f1d5f
>2012-12-17 08:29:18,783 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:29:18,783 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 556f1d5f
>2012-12-17 08:29:18,835 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:29:18,856 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4556
>2012-12-17 08:29:18,858 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4556, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: f4c6985
>2012-12-17 08:29:18,882 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] spmStart polling started: taskId = 8d2c8abb-150d-48c6-b08a-6ef288aef0b0
>2012-12-17 08:29:19,904 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] spmStart polling ended: taskId = 8d2c8abb-150d-48c6-b08a-6ef288aef0b0 task status = finished
>2012-12-17 08:29:19,922 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] spmStart polling ended, spm status: SPM
>2012-12-17 08:29:19,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8d2c8abb-150d-48c6-b08a-6ef288aef0b0), log id: 6533f080
>2012-12-17 08:29:19,943 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, HSMClearTaskVDSCommand, log id: 6533f080
>2012-12-17 08:29:19,943 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@34f29768, log id: f4c6985
>2012-12-17 08:29:20,013 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:29:20,035 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:29:20,039 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 149d45f6
>2012-12-17 08:29:20,058 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 149d45f6
>2012-12-17 08:29:20,058 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 114c1056
>2012-12-17 08:29:20,058 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [54cb0ab7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:29:20,058 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 615c240c
>2012-12-17 08:29:20,058 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:29:20,059 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4447bffd
>2012-12-17 08:29:20,078 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4447bffd
>2012-12-17 08:29:20,078 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 615c240c
>2012-12-17 08:29:20,078 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [54cb0ab7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:29:26,208 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [5863fdac] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:29:28,861 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3bf47b4b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:29:28,998 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [73dddeb9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:29:29,070 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 40c9897d
>2012-12-17 08:29:29,072 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2fa569ea
>2012-12-17 08:29:29,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:29:29,264 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] FINISH, SpmStopVDSCommand, log id: 2fa569ea
>2012-12-17 08:29:29,297 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] FINISH, ResetIrsVDSCommand, log id: 40c9897d
>2012-12-17 08:29:29,308 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 298898f8
>2012-12-17 08:29:29,308 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [73dddeb9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:29:29,308 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 298898f8
>2012-12-17 08:29:36,637 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [69b76134] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:29:39,384 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [17cf9677] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:29:39,399 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [17cf9677] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4557
>2012-12-17 08:29:39,400 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [17cf9677] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4557, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 41fd88e1
>2012-12-17 08:29:39,422 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [17cf9677] spmStart polling started: taskId = fb61d30d-53a2-4a50-8f7a-58bd721059d0
>2012-12-17 08:29:40,445 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [17cf9677] spmStart polling ended: taskId = fb61d30d-53a2-4a50-8f7a-58bd721059d0 task status = finished
>2012-12-17 08:29:40,463 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [17cf9677] spmStart polling ended, spm status: SPM
>2012-12-17 08:29:40,464 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [17cf9677] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=fb61d30d-53a2-4a50-8f7a-58bd721059d0), log id: 4bf777d0
>2012-12-17 08:29:40,482 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [17cf9677] FINISH, HSMClearTaskVDSCommand, log id: 4bf777d0
>2012-12-17 08:29:40,482 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [17cf9677] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@63db1fa6, log id: 41fd88e1
>2012-12-17 08:29:41,317 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [17cf9677] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:29:41,726 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [51b75138] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:29:41,727 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 69267410
>2012-12-17 08:29:41,763 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [17cf9677] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:29:41,942 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [57c34106] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:29:42,045 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [57c34106] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1c428764
>2012-12-17 08:29:42,046 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [57c34106] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5dccff29
>2012-12-17 08:29:42,067 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [57c34106] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:29:42,224 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [57c34106] FINISH, SpmStopVDSCommand, log id: 5dccff29
>2012-12-17 08:29:42,245 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [57c34106] FINISH, ResetIrsVDSCommand, log id: 1c428764
>2012-12-17 08:29:42,256 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [57c34106] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: ea000ed
>2012-12-17 08:29:42,256 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [57c34106] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:29:42,256 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [57c34106] FINISH, MarkPoolInReconstructModeVDSCommand, log id: ea000ed
>2012-12-17 08:29:42,303 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:29:42,317 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4558
>2012-12-17 08:29:42,318 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4558, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 779b4d44
>2012-12-17 08:29:42,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] spmStart polling started: taskId = 217dcb1f-231d-4eed-931a-a9e66ca68ab7
>2012-12-17 08:29:43,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] spmStart polling ended: taskId = 217dcb1f-231d-4eed-931a-a9e66ca68ab7 task status = finished
>2012-12-17 08:29:43,391 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] spmStart polling ended, spm status: SPM
>2012-12-17 08:29:43,392 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=217dcb1f-231d-4eed-931a-a9e66ca68ab7), log id: 348435c0
>2012-12-17 08:29:43,410 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, HSMClearTaskVDSCommand, log id: 348435c0
>2012-12-17 08:29:43,410 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@53e4eb2f, log id: 779b4d44
>2012-12-17 08:29:43,447 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:29:43,491 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:29:43,493 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2491ab1c
>2012-12-17 08:29:43,512 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2491ab1c
>2012-12-17 08:29:43,512 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 69267410
>2012-12-17 08:29:43,512 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [51b75138] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:29:43,512 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3c1dd720
>2012-12-17 08:29:43,512 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:29:43,513 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 786b1035
>2012-12-17 08:29:43,531 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 786b1035
>2012-12-17 08:29:43,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3c1dd720
>2012-12-17 08:29:43,531 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [51b75138] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:29:46,814 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [5b075498] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:29:52,337 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [48110f6e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:29:52,448 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [369a76fa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:29:52,500 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 13ddb57b
>2012-12-17 08:29:52,501 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: afc1624
>2012-12-17 08:29:52,521 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:29:52,672 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] FINISH, SpmStopVDSCommand, log id: afc1624
>2012-12-17 08:29:52,687 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] FINISH, ResetIrsVDSCommand, log id: 13ddb57b
>2012-12-17 08:29:52,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76a97685
>2012-12-17 08:29:52,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [369a76fa] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:29:52,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76a97685
>2012-12-17 08:29:57,028 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:30:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [664081c5] Checking autorecoverable hosts
>2012-12-17 08:30:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [664081c5] Autorecovering 0 hosts
>2012-12-17 08:30:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [664081c5] Checking autorecoverable hosts done
>2012-12-17 08:30:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [664081c5] Autorecovering storage domains is disabled, skipping
>2012-12-17 08:30:02,750 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [376f3052] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:30:02,782 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [376f3052] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4559
>2012-12-17 08:30:02,783 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [376f3052] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4559, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1ac2fbbc
>2012-12-17 08:30:02,805 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [376f3052] spmStart polling started: taskId = 217573cb-1322-4fe6-bf3b-e2a426484469
>2012-12-17 08:30:03,831 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [376f3052] spmStart polling ended: taskId = 217573cb-1322-4fe6-bf3b-e2a426484469 task status = finished
>2012-12-17 08:30:03,850 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [376f3052] spmStart polling ended, spm status: SPM
>2012-12-17 08:30:03,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [376f3052] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=217573cb-1322-4fe6-bf3b-e2a426484469), log id: f69603d
>2012-12-17 08:30:03,869 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [376f3052] FINISH, HSMClearTaskVDSCommand, log id: f69603d
>2012-12-17 08:30:03,869 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [376f3052] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f2d4cb2, log id: 1ac2fbbc
>2012-12-17 08:30:03,885 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [376f3052] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:30:03,917 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-52) [65048227] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:30:03,918 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 223c183
>2012-12-17 08:30:03,943 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [376f3052] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:30:04,097 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [b1109c1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:30:04,179 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [b1109c1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 13b1f5ce
>2012-12-17 08:30:04,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [b1109c1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3d7d70b6
>2012-12-17 08:30:04,200 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [b1109c1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:04,629 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [b1109c1] FINISH, SpmStopVDSCommand, log id: 3d7d70b6
>2012-12-17 08:30:04,643 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [b1109c1] FINISH, ResetIrsVDSCommand, log id: 13b1f5ce
>2012-12-17 08:30:04,655 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [b1109c1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e737678
>2012-12-17 08:30:04,655 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [b1109c1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:04,655 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [b1109c1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e737678
>2012-12-17 08:30:04,705 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:30:04,721 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4560
>2012-12-17 08:30:04,722 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4560, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 69f10d48
>2012-12-17 08:30:04,745 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] spmStart polling started: taskId = a222a6c2-ab4f-49f8-8a64-8339516e2999
>2012-12-17 08:30:05,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] spmStart polling ended: taskId = a222a6c2-ab4f-49f8-8a64-8339516e2999 task status = finished
>2012-12-17 08:30:05,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] spmStart polling ended, spm status: SPM
>2012-12-17 08:30:05,786 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a222a6c2-ab4f-49f8-8a64-8339516e2999), log id: 13051223
>2012-12-17 08:30:05,804 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, HSMClearTaskVDSCommand, log id: 13051223
>2012-12-17 08:30:05,804 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31e9c196, log id: 69f10d48
>2012-12-17 08:30:06,172 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:30:06,245 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:06,248 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 42062e4f
>2012-12-17 08:30:06,269 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 42062e4f
>2012-12-17 08:30:06,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 223c183
>2012-12-17 08:30:06,270 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-52) [65048227] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:30:06,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3d08c4e5
>2012-12-17 08:30:06,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:06,272 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d4004b5
>2012-12-17 08:30:06,290 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d4004b5
>2012-12-17 08:30:06,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3d08c4e5
>2012-12-17 08:30:06,291 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-52) [65048227] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:30:07,238 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b5bb621] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:30:14,744 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [6bdbfd31] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:30:14,841 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [4557a90c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:30:14,915 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [4557a90c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 12c594e9
>2012-12-17 08:30:14,916 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [4557a90c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 68bcf9f3
>2012-12-17 08:30:14,936 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [4557a90c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:15,080 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [4557a90c] FINISH, SpmStopVDSCommand, log id: 68bcf9f3
>2012-12-17 08:30:15,098 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [4557a90c] FINISH, ResetIrsVDSCommand, log id: 12c594e9
>2012-12-17 08:30:15,109 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [4557a90c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 45a97ba2
>2012-12-17 08:30:15,109 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [4557a90c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:15,109 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [4557a90c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 45a97ba2
>2012-12-17 08:30:17,483 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [73dddeb9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:30:25,156 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [152503f5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:30:25,344 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [152503f5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4561
>2012-12-17 08:30:25,346 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [152503f5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4561, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 15f6249b
>2012-12-17 08:30:25,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [152503f5] spmStart polling started: taskId = 82afa13a-e7da-4946-ad42-eba6c560549e
>2012-12-17 08:30:26,389 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [152503f5] spmStart polling ended: taskId = 82afa13a-e7da-4946-ad42-eba6c560549e task status = finished
>2012-12-17 08:30:26,408 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [152503f5] spmStart polling ended, spm status: SPM
>2012-12-17 08:30:26,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [152503f5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=82afa13a-e7da-4946-ad42-eba6c560549e), log id: 235c5b75
>2012-12-17 08:30:26,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [152503f5] FINISH, HSMClearTaskVDSCommand, log id: 235c5b75
>2012-12-17 08:30:26,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [152503f5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4787076, log id: 15f6249b
>2012-12-17 08:30:26,469 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [152503f5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:30:26,525 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [16c2b2c4] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:30:26,525 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 114bf81
>2012-12-17 08:30:26,540 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [152503f5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:30:26,634 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [733172b2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:30:26,751 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6832647e
>2012-12-17 08:30:26,752 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: ab668e1
>2012-12-17 08:30:26,772 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [733172b2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:26,896 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SpmStopVDSCommand, log id: ab668e1
>2012-12-17 08:30:26,911 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, ResetIrsVDSCommand, log id: 6832647e
>2012-12-17 08:30:26,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2221f964
>2012-12-17 08:30:26,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:26,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2221f964
>2012-12-17 08:30:26,974 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [16c2b2c4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:30:26,996 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [16c2b2c4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4562
>2012-12-17 08:30:26,997 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4562, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: b561214
>2012-12-17 08:30:27,020 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] spmStart polling started: taskId = 453e98c1-8acb-4fb8-bdbc-8838936643e3
>2012-12-17 08:30:28,041 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] spmStart polling ended: taskId = 453e98c1-8acb-4fb8-bdbc-8838936643e3 task status = finished
>2012-12-17 08:30:28,059 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] spmStart polling ended, spm status: SPM
>2012-12-17 08:30:28,061 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=453e98c1-8acb-4fb8-bdbc-8838936643e3), log id: b9d0271
>2012-12-17 08:30:28,080 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, HSMClearTaskVDSCommand, log id: b9d0271
>2012-12-17 08:30:28,080 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@710b4eab, log id: b561214
>2012-12-17 08:30:28,091 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [16c2b2c4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:30:28,121 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:28,125 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4911fe04
>2012-12-17 08:30:28,146 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4911fe04
>2012-12-17 08:30:28,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 114bf81
>2012-12-17 08:30:28,146 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [16c2b2c4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:30:28,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7e71953e
>2012-12-17 08:30:28,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:28,147 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 64f70287
>2012-12-17 08:30:28,166 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 64f70287
>2012-12-17 08:30:28,166 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7e71953e
>2012-12-17 08:30:28,166 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [16c2b2c4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:30:37,051 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [55210d2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:30:37,176 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [169ff372] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:30:37,224 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 8033687
>2012-12-17 08:30:37,226 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: bdcb13e
>2012-12-17 08:30:37,245 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [169ff372] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:37,388 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, SpmStopVDSCommand, log id: bdcb13e
>2012-12-17 08:30:37,423 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, ResetIrsVDSCommand, log id: 8033687
>2012-12-17 08:30:37,434 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 534af002
>2012-12-17 08:30:37,434 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:37,434 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 534af002
>2012-12-17 08:30:37,879 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3e6ad503] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:30:47,495 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [7bc37769] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:30:47,515 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [7bc37769] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4563
>2012-12-17 08:30:47,516 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4563, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1be93b3e
>2012-12-17 08:30:47,542 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] spmStart polling started: taskId = bb692280-bb1b-46cc-a7bc-9f39f57063bf
>2012-12-17 08:30:48,563 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] spmStart polling ended: taskId = bb692280-bb1b-46cc-a7bc-9f39f57063bf task status = finished
>2012-12-17 08:30:48,583 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] spmStart polling ended, spm status: SPM
>2012-12-17 08:30:48,584 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bb692280-bb1b-46cc-a7bc-9f39f57063bf), log id: 53b84286
>2012-12-17 08:30:48,602 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] FINISH, HSMClearTaskVDSCommand, log id: 53b84286
>2012-12-17 08:30:48,602 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6617965a, log id: 1be93b3e
>2012-12-17 08:30:48,625 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [7bc37769] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:30:48,689 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [67020b95] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:30:48,689 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 609d6f9a
>2012-12-17 08:30:48,721 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [7bc37769] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:30:48,893 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [181d86f1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:30:48,987 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 55c26eba
>2012-12-17 08:30:48,990 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1554053b
>2012-12-17 08:30:49,014 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:49,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SpmStopVDSCommand, log id: 1554053b
>2012-12-17 08:30:49,726 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, ResetIrsVDSCommand, log id: 55c26eba
>2012-12-17 08:30:49,763 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: d729433
>2012-12-17 08:30:49,763 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:49,763 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: d729433
>2012-12-17 08:30:49,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:30:50,007 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4564
>2012-12-17 08:30:50,008 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4564, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4d29fbd8
>2012-12-17 08:30:50,038 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] spmStart polling started: taskId = 1ed39c45-91ff-4337-82c2-75af88e9146c
>2012-12-17 08:30:51,060 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] spmStart polling ended: taskId = 1ed39c45-91ff-4337-82c2-75af88e9146c task status = finished
>2012-12-17 08:30:51,079 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] spmStart polling ended, spm status: SPM
>2012-12-17 08:30:51,080 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1ed39c45-91ff-4337-82c2-75af88e9146c), log id: 66646150
>2012-12-17 08:30:51,099 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, HSMClearTaskVDSCommand, log id: 66646150
>2012-12-17 08:30:51,099 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4830ffb4, log id: 4d29fbd8
>2012-12-17 08:30:51,456 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:30:51,569 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:51,572 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 17ab2de8
>2012-12-17 08:30:51,594 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 17ab2de8
>2012-12-17 08:30:51,594 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 609d6f9a
>2012-12-17 08:30:51,594 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [67020b95] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:30:51,594 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7399ca3b
>2012-12-17 08:30:51,594 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:30:51,596 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 110e1121
>2012-12-17 08:30:51,614 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 110e1121
>2012-12-17 08:30:51,614 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7399ca3b
>2012-12-17 08:30:51,614 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [67020b95] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:30:58,233 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [35e0a925] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:30:59,986 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [57b06faf] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:31:00,257 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [69e82ff1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:31:01,086 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 122a3a39
>2012-12-17 08:31:01,088 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 43e23c4b
>2012-12-17 08:31:01,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:31:01,294 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] FINISH, SpmStopVDSCommand, log id: 43e23c4b
>2012-12-17 08:31:01,306 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] FINISH, ResetIrsVDSCommand, log id: 122a3a39
>2012-12-17 08:31:01,317 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7bda5372
>2012-12-17 08:31:01,317 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69e82ff1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:31:01,317 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7bda5372
>2012-12-17 08:31:08,472 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [f50ed1e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:31:11,379 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [53e8b92c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:31:11,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [53e8b92c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4565
>2012-12-17 08:31:11,486 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4565, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 30ca9c00
>2012-12-17 08:31:11,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] spmStart polling started: taskId = 2e43d571-fb34-4d9a-9d9f-b7b60ea6256d
>2012-12-17 08:31:12,561 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] spmStart polling ended: taskId = 2e43d571-fb34-4d9a-9d9f-b7b60ea6256d task status = finished
>2012-12-17 08:31:12,592 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] spmStart polling ended, spm status: SPM
>2012-12-17 08:31:12,594 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2e43d571-fb34-4d9a-9d9f-b7b60ea6256d), log id: 22a84930
>2012-12-17 08:31:12,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] FINISH, HSMClearTaskVDSCommand, log id: 22a84930
>2012-12-17 08:31:12,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@565d0016, log id: 30ca9c00
>2012-12-17 08:31:12,641 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [53e8b92c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:31:12,693 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [65976fd0] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:31:12,693 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 24a70eff
>2012-12-17 08:31:12,712 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [53e8b92c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:31:12,899 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [57e0db40] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:31:12,964 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: bc8e6d2
>2012-12-17 08:31:12,965 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2d03bc1d
>2012-12-17 08:31:12,985 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:31:13,463 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] FINISH, SpmStopVDSCommand, log id: 2d03bc1d
>2012-12-17 08:31:13,501 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] FINISH, ResetIrsVDSCommand, log id: bc8e6d2
>2012-12-17 08:31:13,512 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5bf14db5
>2012-12-17 08:31:13,512 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [57e0db40] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:31:13,512 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5bf14db5
>2012-12-17 08:31:13,656 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [65976fd0] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:31:13,718 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [65976fd0] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4566
>2012-12-17 08:31:13,720 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4566, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 16d960f
>2012-12-17 08:31:13,745 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] spmStart polling started: taskId = be25005c-041d-45bf-90cd-11bad888cc93
>2012-12-17 08:31:14,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] spmStart polling ended: taskId = be25005c-041d-45bf-90cd-11bad888cc93 task status = finished
>2012-12-17 08:31:14,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] spmStart polling ended, spm status: SPM
>2012-12-17 08:31:14,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=be25005c-041d-45bf-90cd-11bad888cc93), log id: 6c3457f1
>2012-12-17 08:31:14,804 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, HSMClearTaskVDSCommand, log id: 6c3457f1
>2012-12-17 08:31:14,804 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6079c6b6, log id: 16d960f
>2012-12-17 08:31:14,841 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [65976fd0] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:31:14,864 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:31:14,868 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6457e629
>2012-12-17 08:31:14,887 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6457e629
>2012-12-17 08:31:14,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 24a70eff
>2012-12-17 08:31:14,887 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [65976fd0] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:31:14,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 46c7c136
>2012-12-17 08:31:14,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:31:14,888 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 655a6acb
>2012-12-17 08:31:14,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 655a6acb
>2012-12-17 08:31:14,908 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 46c7c136
>2012-12-17 08:31:14,908 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [65976fd0] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:31:18,742 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:31:23,695 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [311c1434] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:31:23,873 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [f91baee] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:31:23,946 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [f91baee] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2909d058
>2012-12-17 08:31:23,947 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [f91baee] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 76220ced
>2012-12-17 08:31:23,967 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [f91baee] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:31:24,191 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [f91baee] FINISH, SpmStopVDSCommand, log id: 76220ced
>2012-12-17 08:31:24,232 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [f91baee] FINISH, ResetIrsVDSCommand, log id: 2909d058
>2012-12-17 08:31:24,243 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [f91baee] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3ceddd9b
>2012-12-17 08:31:24,243 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [f91baee] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:31:24,243 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [f91baee] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3ceddd9b
>2012-12-17 08:31:29,190 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2c13a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:31:34,333 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1a53ba46] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:31:34,355 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1a53ba46] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4567
>2012-12-17 08:31:34,356 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1a53ba46] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4567, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 35a2d58e
>2012-12-17 08:31:34,377 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1a53ba46] spmStart polling started: taskId = 28958245-1f85-4593-99f3-adf6374a61af
>2012-12-17 08:31:35,398 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1a53ba46] spmStart polling ended: taskId = 28958245-1f85-4593-99f3-adf6374a61af task status = finished
>2012-12-17 08:31:35,416 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1a53ba46] spmStart polling ended, spm status: SPM
>2012-12-17 08:31:35,417 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [1a53ba46] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=28958245-1f85-4593-99f3-adf6374a61af), log id: 3a087a18
>2012-12-17 08:31:35,435 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [1a53ba46] FINISH, HSMClearTaskVDSCommand, log id: 3a087a18
>2012-12-17 08:31:35,436 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1a53ba46] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@287db6c, log id: 35a2d58e
>2012-12-17 08:31:35,780 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1a53ba46] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:31:36,414 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [2e21b599] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:31:36,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 606f7d3c
>2012-12-17 08:31:36,447 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1a53ba46] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:31:36,623 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-26) [27ff9aa4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:31:36,685 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [27ff9aa4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 458c9c83
>2012-12-17 08:31:36,686 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [27ff9aa4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 216e500c
>2012-12-17 08:31:36,706 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [27ff9aa4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:31:36,833 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [27ff9aa4] FINISH, SpmStopVDSCommand, log id: 216e500c
>2012-12-17 08:31:36,851 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [27ff9aa4] FINISH, ResetIrsVDSCommand, log id: 458c9c83
>2012-12-17 08:31:36,862 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [27ff9aa4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: b8eb76d
>2012-12-17 08:31:36,862 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [27ff9aa4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:31:36,862 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [27ff9aa4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: b8eb76d
>2012-12-17 08:31:36,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:31:36,965 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4568
>2012-12-17 08:31:36,966 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4568, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: ceff4fa
>2012-12-17 08:31:36,987 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] spmStart polling started: taskId = d72d3eec-fac8-4bba-8176-14d19152d16e
>2012-12-17 08:31:38,009 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] spmStart polling ended: taskId = d72d3eec-fac8-4bba-8176-14d19152d16e task status = finished
>2012-12-17 08:31:38,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] spmStart polling ended, spm status: SPM
>2012-12-17 08:31:38,028 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d72d3eec-fac8-4bba-8176-14d19152d16e), log id: 67aeda9e
>2012-12-17 08:31:38,046 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, HSMClearTaskVDSCommand, log id: 67aeda9e
>2012-12-17 08:31:38,046 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@892f040, log id: ceff4fa
>2012-12-17 08:31:38,068 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:31:38,101 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:31:38,105 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4f5ee3b9
>2012-12-17 08:31:38,125 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4f5ee3b9
>2012-12-17 08:31:38,125 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 606f7d3c
>2012-12-17 08:31:38,125 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [2e21b599] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:31:38,125 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5eb8af64
>2012-12-17 08:31:38,125 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:31:38,126 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 66b9a70c
>2012-12-17 08:31:38,144 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 66b9a70c
>2012-12-17 08:31:38,144 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5eb8af64
>2012-12-17 08:31:38,144 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [2e21b599] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:31:39,409 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [323f195a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:31:46,974 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [35e0a925] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:31:47,104 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-62) [2edbd860] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:31:47,179 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 768b098a
>2012-12-17 08:31:47,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6defbb08
>2012-12-17 08:31:47,201 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:31:47,936 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, SpmStopVDSCommand, log id: 6defbb08
>2012-12-17 08:31:47,954 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, ResetIrsVDSCommand, log id: 768b098a
>2012-12-17 08:31:47,966 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 78c751c8
>2012-12-17 08:31:47,966 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2edbd860] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:31:47,966 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 78c751c8
>2012-12-17 08:31:49,763 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [4557a90c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:31:58,018 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [69b76134] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:31:58,059 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [69b76134] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4569
>2012-12-17 08:31:58,061 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [69b76134] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4569, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7b61ee0a
>2012-12-17 08:31:58,083 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [69b76134] spmStart polling started: taskId = d3593a90-81f4-49a8-a78e-b153569d87db
>2012-12-17 08:31:59,107 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [69b76134] spmStart polling ended: taskId = d3593a90-81f4-49a8-a78e-b153569d87db task status = finished
>2012-12-17 08:31:59,127 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [69b76134] spmStart polling ended, spm status: SPM
>2012-12-17 08:31:59,129 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [69b76134] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d3593a90-81f4-49a8-a78e-b153569d87db), log id: 61bc802e
>2012-12-17 08:31:59,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [69b76134] FINISH, HSMClearTaskVDSCommand, log id: 61bc802e
>2012-12-17 08:31:59,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [69b76134] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1e837e23, log id: 7b61ee0a
>2012-12-17 08:31:59,164 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [69b76134] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:31:59,253 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:31:59,253 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3929ad27
>2012-12-17 08:31:59,272 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [69b76134] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:31:59,406 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [6f019f85] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:31:59,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5e59089d
>2012-12-17 08:31:59,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7b3af3f9
>2012-12-17 08:31:59,495 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:00,008 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7f13ba47] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:32:00,300 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] FINISH, SpmStopVDSCommand, log id: 7b3af3f9
>2012-12-17 08:32:01,115 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] FINISH, ResetIrsVDSCommand, log id: 5e59089d
>2012-12-17 08:32:01,126 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7fd3326c
>2012-12-17 08:32:01,126 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [6f019f85] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:01,126 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7fd3326c
>2012-12-17 08:32:01,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:32:01,345 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4570
>2012-12-17 08:32:01,347 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4570, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 700fcf5e
>2012-12-17 08:32:01,371 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling started: taskId = e04167e8-338a-4376-a30b-b01cd102e874
>2012-12-17 08:32:02,392 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended: taskId = e04167e8-338a-4376-a30b-b01cd102e874 task status = finished
>2012-12-17 08:32:02,410 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended, spm status: SPM
>2012-12-17 08:32:02,411 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e04167e8-338a-4376-a30b-b01cd102e874), log id: 6bb7ab1f
>2012-12-17 08:32:02,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMClearTaskVDSCommand, log id: 6bb7ab1f
>2012-12-17 08:32:02,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6743b778, log id: 700fcf5e
>2012-12-17 08:32:02,444 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:32:02,461 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:02,463 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 74c7161b
>2012-12-17 08:32:02,482 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 74c7161b
>2012-12-17 08:32:02,482 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3929ad27
>2012-12-17 08:32:02,482 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:32:02,482 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3a7c8282
>2012-12-17 08:32:02,482 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:02,483 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: c4307ea
>2012-12-17 08:32:02,502 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: c4307ea
>2012-12-17 08:32:02,502 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3a7c8282
>2012-12-17 08:32:02,502 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:32:02,521 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [436410e6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:32:02,521 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7a1e5dfd
>2012-12-17 08:32:02,521 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:02,523 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3e6e0e5e
>2012-12-17 08:32:02,541 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3e6e0e5e
>2012-12-17 08:32:02,541 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7a1e5dfd
>2012-12-17 08:32:02,541 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [436410e6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:32:02,541 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2b715d57
>2012-12-17 08:32:02,541 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:02,542 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 379a21f4
>2012-12-17 08:32:02,560 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 379a21f4
>2012-12-17 08:32:02,561 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2b715d57
>2012-12-17 08:32:02,561 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [436410e6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:32:11,386 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1eda0292] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:32:11,662 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [70373480] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:32:11,798 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [70373480] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 54c8952c
>2012-12-17 08:32:11,799 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [70373480] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6d546d52
>2012-12-17 08:32:11,821 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [70373480] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:12,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [70373480] FINISH, SpmStopVDSCommand, log id: 6d546d52
>2012-12-17 08:32:12,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [70373480] FINISH, ResetIrsVDSCommand, log id: 54c8952c
>2012-12-17 08:32:12,131 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [70373480] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5466df99
>2012-12-17 08:32:12,131 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [70373480] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:12,131 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [70373480] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5466df99
>2012-12-17 08:32:21,008 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:32:22,244 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [664081c5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:32:22,307 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [664081c5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4571
>2012-12-17 08:32:22,310 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [664081c5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4571, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4dd03bec
>2012-12-17 08:32:22,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [664081c5] spmStart polling started: taskId = ddfd3ff5-6724-43e7-bfe0-fb09f4a920dc
>2012-12-17 08:32:23,355 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [664081c5] spmStart polling ended: taskId = ddfd3ff5-6724-43e7-bfe0-fb09f4a920dc task status = finished
>2012-12-17 08:32:23,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [664081c5] spmStart polling ended, spm status: SPM
>2012-12-17 08:32:23,377 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [664081c5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ddfd3ff5-6724-43e7-bfe0-fb09f4a920dc), log id: 47755e2a
>2012-12-17 08:32:23,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [664081c5] FINISH, HSMClearTaskVDSCommand, log id: 47755e2a
>2012-12-17 08:32:23,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [664081c5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a793c0c, log id: 4dd03bec
>2012-12-17 08:32:23,432 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [664081c5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:32:23,507 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [181d86f1] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:32:23,507 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5baf5588
>2012-12-17 08:32:23,547 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [664081c5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:32:23,704 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-12) [2f94bcb4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:32:23,757 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 23916919
>2012-12-17 08:32:23,759 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 57791c21
>2012-12-17 08:32:23,779 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:23,917 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, SpmStopVDSCommand, log id: 57791c21
>2012-12-17 08:32:23,995 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, ResetIrsVDSCommand, log id: 23916919
>2012-12-17 08:32:24,005 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7cc4080d
>2012-12-17 08:32:24,006 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2f94bcb4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:24,006 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7cc4080d
>2012-12-17 08:32:24,119 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:32:24,139 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4572
>2012-12-17 08:32:24,141 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4572, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2b50e055
>2012-12-17 08:32:24,163 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling started: taskId = 7863d8ea-ba19-4738-8d6d-eaf254a1d9ed
>2012-12-17 08:32:25,184 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling ended: taskId = 7863d8ea-ba19-4738-8d6d-eaf254a1d9ed task status = finished
>2012-12-17 08:32:25,202 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling ended, spm status: SPM
>2012-12-17 08:32:25,203 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7863d8ea-ba19-4738-8d6d-eaf254a1d9ed), log id: 2b948005
>2012-12-17 08:32:25,221 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, HSMClearTaskVDSCommand, log id: 2b948005
>2012-12-17 08:32:25,221 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2260de9d, log id: 2b50e055
>2012-12-17 08:32:25,258 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:32:25,657 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:25,660 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7c07758e
>2012-12-17 08:32:25,682 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7c07758e
>2012-12-17 08:32:25,682 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5baf5588
>2012-12-17 08:32:25,682 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [181d86f1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:32:25,682 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4db6fd3f
>2012-12-17 08:32:25,682 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:25,683 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6f1bb006
>2012-12-17 08:32:25,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f1bb006
>2012-12-17 08:32:25,702 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4db6fd3f
>2012-12-17 08:32:25,702 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [181d86f1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:32:31,301 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:32:34,217 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [3ff768c3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:32:34,324 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [69d0a8ae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:32:34,393 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2a8b02d8
>2012-12-17 08:32:34,394 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5766d89a
>2012-12-17 08:32:34,415 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:35,135 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] FINISH, SpmStopVDSCommand, log id: 5766d89a
>2012-12-17 08:32:35,155 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] FINISH, ResetIrsVDSCommand, log id: 2a8b02d8
>2012-12-17 08:32:35,166 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: b4180b
>2012-12-17 08:32:35,166 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [69d0a8ae] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:35,166 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: b4180b
>2012-12-17 08:32:41,809 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:32:45,266 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [f50ed1e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:32:45,385 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [f50ed1e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4573
>2012-12-17 08:32:45,387 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [f50ed1e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4573, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5609ce79
>2012-12-17 08:32:45,408 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [f50ed1e] spmStart polling started: taskId = 649c9b80-83d5-4286-b69e-979de9adb8f5
>2012-12-17 08:32:46,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [f50ed1e] spmStart polling ended: taskId = 649c9b80-83d5-4286-b69e-979de9adb8f5 task status = finished
>2012-12-17 08:32:46,448 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [f50ed1e] spmStart polling ended, spm status: SPM
>2012-12-17 08:32:46,450 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [f50ed1e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=649c9b80-83d5-4286-b69e-979de9adb8f5), log id: 1e674106
>2012-12-17 08:32:46,469 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [f50ed1e] FINISH, HSMClearTaskVDSCommand, log id: 1e674106
>2012-12-17 08:32:46,469 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [f50ed1e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@cde0082, log id: 5609ce79
>2012-12-17 08:32:46,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [f50ed1e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:32:46,770 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [495d106] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:32:46,770 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 62f8dfcc
>2012-12-17 08:32:46,788 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [f50ed1e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:32:46,977 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [48f6cd0b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:32:47,115 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [48f6cd0b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 56430a0f
>2012-12-17 08:32:47,116 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [48f6cd0b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6e2dea80
>2012-12-17 08:32:47,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [48f6cd0b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:47,295 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [48f6cd0b] FINISH, SpmStopVDSCommand, log id: 6e2dea80
>2012-12-17 08:32:47,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [48f6cd0b] FINISH, ResetIrsVDSCommand, log id: 56430a0f
>2012-12-17 08:32:47,326 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [48f6cd0b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bee922
>2012-12-17 08:32:47,326 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [48f6cd0b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:47,326 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [48f6cd0b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bee922
>2012-12-17 08:32:47,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [495d106] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:32:47,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [495d106] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4574
>2012-12-17 08:32:47,393 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4574, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3169d127
>2012-12-17 08:32:47,416 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [495d106] spmStart polling started: taskId = dfa6a6ef-692b-4117-8e15-af5eec87f06d
>2012-12-17 08:32:48,437 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [495d106] spmStart polling ended: taskId = dfa6a6ef-692b-4117-8e15-af5eec87f06d task status = finished
>2012-12-17 08:32:48,456 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [495d106] spmStart polling ended, spm status: SPM
>2012-12-17 08:32:48,457 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dfa6a6ef-692b-4117-8e15-af5eec87f06d), log id: 10978e8c
>2012-12-17 08:32:48,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, HSMClearTaskVDSCommand, log id: 10978e8c
>2012-12-17 08:32:48,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4e0f91d6, log id: 3169d127
>2012-12-17 08:32:48,495 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [495d106] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:32:48,535 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:48,537 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2c19855f
>2012-12-17 08:32:48,558 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2c19855f
>2012-12-17 08:32:48,558 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 62f8dfcc
>2012-12-17 08:32:48,558 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [495d106] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:32:48,558 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3f833697
>2012-12-17 08:32:48,558 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:48,560 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3471569e
>2012-12-17 08:32:48,578 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3471569e
>2012-12-17 08:32:48,578 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f833697
>2012-12-17 08:32:48,578 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [495d106] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:32:52,088 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [57c34106] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:32:57,433 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [4f7e897c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:32:57,698 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-97) [7b0d0943] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:32:57,788 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: d0d6178
>2012-12-17 08:32:57,789 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 31ff602
>2012-12-17 08:32:57,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:58,432 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SpmStopVDSCommand, log id: 31ff602
>2012-12-17 08:32:58,488 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, ResetIrsVDSCommand, log id: d0d6178
>2012-12-17 08:32:58,499 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2892c54f
>2012-12-17 08:32:58,499 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:32:58,499 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2892c54f
>2012-12-17 08:33:02,301 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:33:08,609 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [4adae006] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:33:08,663 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [4adae006] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4575
>2012-12-17 08:33:08,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [4adae006] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4575, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5812546
>2012-12-17 08:33:08,686 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [4adae006] spmStart polling started: taskId = 7e1bc965-cae2-4feb-8dc5-1ba5b90a996f
>2012-12-17 08:33:09,709 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [4adae006] spmStart polling ended: taskId = 7e1bc965-cae2-4feb-8dc5-1ba5b90a996f task status = finished
>2012-12-17 08:33:09,727 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [4adae006] spmStart polling ended, spm status: SPM
>2012-12-17 08:33:09,728 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [4adae006] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7e1bc965-cae2-4feb-8dc5-1ba5b90a996f), log id: 76299571
>2012-12-17 08:33:09,747 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [4adae006] FINISH, HSMClearTaskVDSCommand, log id: 76299571
>2012-12-17 08:33:09,747 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [4adae006] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@24126a52, log id: 5812546
>2012-12-17 08:33:09,759 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [4adae006] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:33:09,799 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [5ab536e] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:33:09,799 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7785b9b8
>2012-12-17 08:33:09,819 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [4adae006] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:33:09,941 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-29) [ee8ec0b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:33:10,003 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5e5a71a4
>2012-12-17 08:33:10,004 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 69ef467e
>2012-12-17 08:33:10,024 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:10,538 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] FINISH, SpmStopVDSCommand, log id: 69ef467e
>2012-12-17 08:33:11,068 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] FINISH, ResetIrsVDSCommand, log id: 5e5a71a4
>2012-12-17 08:33:11,079 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4f56ec7e
>2012-12-17 08:33:11,080 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:11,080 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4f56ec7e
>2012-12-17 08:33:11,284 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:33:11,312 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4576
>2012-12-17 08:33:11,314 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4576, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3c1bdff2
>2012-12-17 08:33:11,336 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] spmStart polling started: taskId = aa93f088-586b-46a2-8a46-2871b06435c3
>2012-12-17 08:33:12,359 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] spmStart polling ended: taskId = aa93f088-586b-46a2-8a46-2871b06435c3 task status = finished
>2012-12-17 08:33:12,378 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] spmStart polling ended, spm status: SPM
>2012-12-17 08:33:12,379 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=aa93f088-586b-46a2-8a46-2871b06435c3), log id: 3377cf24
>2012-12-17 08:33:12,397 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, HSMClearTaskVDSCommand, log id: 3377cf24
>2012-12-17 08:33:12,397 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7ec05ee1, log id: 3c1bdff2
>2012-12-17 08:33:12,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:33:12,442 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:12,443 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: bcaa21a
>2012-12-17 08:33:12,478 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: bcaa21a
>2012-12-17 08:33:12,478 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7785b9b8
>2012-12-17 08:33:12,478 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [5ab536e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:33:12,478 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 24585f02
>2012-12-17 08:33:12,478 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:12,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1ff3349c
>2012-12-17 08:33:12,521 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ff3349c
>2012-12-17 08:33:12,521 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 24585f02
>2012-12-17 08:33:12,521 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [5ab536e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:33:12,621 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4579c6a9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:33:21,461 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [58c4b5b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:33:21,831 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [46701658] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:33:21,935 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 74fb834a
>2012-12-17 08:33:21,937 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 57a4de45
>2012-12-17 08:33:21,961 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [46701658] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:22,107 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, SpmStopVDSCommand, log id: 57a4de45
>2012-12-17 08:33:22,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, ResetIrsVDSCommand, log id: 74fb834a
>2012-12-17 08:33:22,163 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bcbe5fb
>2012-12-17 08:33:22,163 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:22,163 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bcbe5fb
>2012-12-17 08:33:22,889 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:33:32,228 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [a209fdc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:33:32,284 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [a209fdc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4577
>2012-12-17 08:33:32,285 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4577, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 28e3af35
>2012-12-17 08:33:32,307 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] spmStart polling started: taskId = 1dc1d66c-7eee-422d-8a30-6246a1a9de9f
>2012-12-17 08:33:33,328 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] spmStart polling ended: taskId = 1dc1d66c-7eee-422d-8a30-6246a1a9de9f task status = finished
>2012-12-17 08:33:33,347 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] spmStart polling ended, spm status: SPM
>2012-12-17 08:33:33,348 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1dc1d66c-7eee-422d-8a30-6246a1a9de9f), log id: 5281e7ec
>2012-12-17 08:33:33,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] FINISH, HSMClearTaskVDSCommand, log id: 5281e7ec
>2012-12-17 08:33:33,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4ea16145, log id: 28e3af35
>2012-12-17 08:33:33,451 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [a209fdc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:33:33,572 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [676a97fd] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:33:33,572 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4cb16a82
>2012-12-17 08:33:33,606 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [a209fdc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:33:33,797 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [47f0782e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:33:33,843 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [47f0782e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1743c19e
>2012-12-17 08:33:33,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [47f0782e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 369be0bb
>2012-12-17 08:33:33,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [47f0782e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:34,263 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [47f0782e] FINISH, SpmStopVDSCommand, log id: 369be0bb
>2012-12-17 08:33:34,314 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [47f0782e] FINISH, ResetIrsVDSCommand, log id: 1743c19e
>2012-12-17 08:33:34,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [47f0782e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49ac0c7f
>2012-12-17 08:33:34,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [47f0782e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:34,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [47f0782e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49ac0c7f
>2012-12-17 08:33:34,511 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [676a97fd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:33:34,527 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [676a97fd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4578
>2012-12-17 08:33:34,528 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4578, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 18caed9f
>2012-12-17 08:33:34,551 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] spmStart polling started: taskId = 9fc2c21d-14e5-4410-875e-a6a49d4d27d1
>2012-12-17 08:33:35,572 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] spmStart polling ended: taskId = 9fc2c21d-14e5-4410-875e-a6a49d4d27d1 task status = finished
>2012-12-17 08:33:35,590 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] spmStart polling ended, spm status: SPM
>2012-12-17 08:33:35,592 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9fc2c21d-14e5-4410-875e-a6a49d4d27d1), log id: 188a4b74
>2012-12-17 08:33:35,610 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, HSMClearTaskVDSCommand, log id: 188a4b74
>2012-12-17 08:33:35,610 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1f5fb942, log id: 18caed9f
>2012-12-17 08:33:35,947 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [676a97fd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:33:36,142 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:36,143 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 9bb4fb1
>2012-12-17 08:33:36,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 9bb4fb1
>2012-12-17 08:33:36,165 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4cb16a82
>2012-12-17 08:33:36,165 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [676a97fd] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:33:36,165 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 686cf274
>2012-12-17 08:33:36,165 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:36,166 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 51299f41
>2012-12-17 08:33:36,184 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 51299f41
>2012-12-17 08:33:36,184 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 686cf274
>2012-12-17 08:33:36,184 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [676a97fd] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:33:43,521 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:33:44,563 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [5b075498] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:33:44,675 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-100) [95f5063] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:33:44,722 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1d3c3e15
>2012-12-17 08:33:44,724 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6ad320c7
>2012-12-17 08:33:44,744 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [95f5063] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:44,902 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, SpmStopVDSCommand, log id: 6ad320c7
>2012-12-17 08:33:44,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, ResetIrsVDSCommand, log id: 1d3c3e15
>2012-12-17 08:33:44,942 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d973f94
>2012-12-17 08:33:44,942 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [95f5063] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:44,942 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d973f94
>2012-12-17 08:33:53,846 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1981e345] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:33:54,998 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [36fcbeec] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:33:55,026 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [36fcbeec] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4579
>2012-12-17 08:33:55,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4579, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 644da0c1
>2012-12-17 08:33:55,050 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] spmStart polling started: taskId = b1e20c66-2f90-43fa-8ef2-b58a3b9a6fec
>2012-12-17 08:33:56,072 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] spmStart polling ended: taskId = b1e20c66-2f90-43fa-8ef2-b58a3b9a6fec task status = finished
>2012-12-17 08:33:56,090 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] spmStart polling ended, spm status: SPM
>2012-12-17 08:33:56,091 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b1e20c66-2f90-43fa-8ef2-b58a3b9a6fec), log id: 4673fb46
>2012-12-17 08:33:56,111 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] FINISH, HSMClearTaskVDSCommand, log id: 4673fb46
>2012-12-17 08:33:56,111 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6875d4cc, log id: 644da0c1
>2012-12-17 08:33:56,126 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [36fcbeec] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:33:56,185 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [51f97e37] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:33:56,185 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 92646a2
>2012-12-17 08:33:56,208 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [36fcbeec] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:33:56,344 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [5b666f42] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:33:56,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4c5e1c10
>2012-12-17 08:33:56,400 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 46843994
>2012-12-17 08:33:56,420 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:56,573 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] FINISH, SpmStopVDSCommand, log id: 46843994
>2012-12-17 08:33:56,609 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] FINISH, ResetIrsVDSCommand, log id: 4c5e1c10
>2012-12-17 08:33:56,620 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 767096ea
>2012-12-17 08:33:56,620 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [5b666f42] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:56,620 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 767096ea
>2012-12-17 08:33:56,686 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:33:56,700 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4580
>2012-12-17 08:33:56,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4580, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44481305
>2012-12-17 08:33:56,724 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] spmStart polling started: taskId = a1b20d7f-1eca-493c-9595-59836d6d3d90
>2012-12-17 08:33:57,747 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] spmStart polling ended: taskId = a1b20d7f-1eca-493c-9595-59836d6d3d90 task status = finished
>2012-12-17 08:33:57,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] spmStart polling ended, spm status: SPM
>2012-12-17 08:33:57,767 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a1b20d7f-1eca-493c-9595-59836d6d3d90), log id: 6a5dfc75
>2012-12-17 08:33:57,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, HSMClearTaskVDSCommand, log id: 6a5dfc75
>2012-12-17 08:33:57,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@40699be1, log id: 44481305
>2012-12-17 08:33:57,836 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:33:57,877 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:57,880 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 56669bf8
>2012-12-17 08:33:57,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 56669bf8
>2012-12-17 08:33:57,901 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 92646a2
>2012-12-17 08:33:57,901 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [51f97e37] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:33:57,901 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6a89e318
>2012-12-17 08:33:57,901 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:33:57,902 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 30ffb8fd
>2012-12-17 08:33:57,920 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 30ffb8fd
>2012-12-17 08:33:57,920 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6a89e318
>2012-12-17 08:33:57,921 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [51f97e37] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:34:04,103 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:34:06,722 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [550b34b1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:34:06,860 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [44e11955] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:34:06,919 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [44e11955] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 64534e44
>2012-12-17 08:34:06,920 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [44e11955] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2d4aa5a1
>2012-12-17 08:34:06,940 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [44e11955] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:34:07,093 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [44e11955] FINISH, SpmStopVDSCommand, log id: 2d4aa5a1
>2012-12-17 08:34:07,118 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [44e11955] FINISH, ResetIrsVDSCommand, log id: 64534e44
>2012-12-17 08:34:07,128 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [44e11955] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 252600e4
>2012-12-17 08:34:07,129 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [44e11955] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:34:07,129 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [44e11955] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 252600e4
>2012-12-17 08:34:14,748 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [4557a90c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:34:17,220 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [267551eb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:34:17,234 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [267551eb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4581
>2012-12-17 08:34:17,236 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [267551eb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4581, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 38700870
>2012-12-17 08:34:17,258 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [267551eb] spmStart polling started: taskId = d1cf5c6c-4d06-41ec-8e0b-76cd1e30d98f
>2012-12-17 08:34:18,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [267551eb] spmStart polling ended: taskId = d1cf5c6c-4d06-41ec-8e0b-76cd1e30d98f task status = finished
>2012-12-17 08:34:18,300 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [267551eb] spmStart polling ended, spm status: SPM
>2012-12-17 08:34:18,301 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [267551eb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d1cf5c6c-4d06-41ec-8e0b-76cd1e30d98f), log id: 5d724dd0
>2012-12-17 08:34:18,320 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [267551eb] FINISH, HSMClearTaskVDSCommand, log id: 5d724dd0
>2012-12-17 08:34:18,320 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [267551eb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7b52c29d, log id: 38700870
>2012-12-17 08:34:18,360 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [267551eb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:34:18,422 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [28319bba] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:34:18,422 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 23f4b03e
>2012-12-17 08:34:18,449 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [267551eb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:34:18,668 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-92) [5fbba580] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:34:18,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 67e8911
>2012-12-17 08:34:18,835 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6765964b
>2012-12-17 08:34:18,854 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:34:19,350 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] FINISH, SpmStopVDSCommand, log id: 6765964b
>2012-12-17 08:34:19,377 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] FINISH, ResetIrsVDSCommand, log id: 67e8911
>2012-12-17 08:34:19,388 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 519d812b
>2012-12-17 08:34:19,388 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [5fbba580] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:34:19,388 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 519d812b
>2012-12-17 08:34:19,472 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:34:19,517 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4582
>2012-12-17 08:34:19,518 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4582, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2d54d0
>2012-12-17 08:34:19,541 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling started: taskId = 26c71c62-1850-4ffe-b7e4-d4e5b7b98eb4
>2012-12-17 08:34:20,562 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling ended: taskId = 26c71c62-1850-4ffe-b7e4-d4e5b7b98eb4 task status = finished
>2012-12-17 08:34:20,584 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling ended, spm status: SPM
>2012-12-17 08:34:20,585 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=26c71c62-1850-4ffe-b7e4-d4e5b7b98eb4), log id: 19c49a18
>2012-12-17 08:34:20,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, HSMClearTaskVDSCommand, log id: 19c49a18
>2012-12-17 08:34:20,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3a60aadf, log id: 2d54d0
>2012-12-17 08:34:21,107 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:34:21,686 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:34:21,688 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6c55f486
>2012-12-17 08:34:21,710 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6c55f486
>2012-12-17 08:34:21,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 23f4b03e
>2012-12-17 08:34:21,711 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [28319bba] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:34:21,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 72cc3a29
>2012-12-17 08:34:21,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:34:21,712 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: b2d2c8b
>2012-12-17 08:34:21,731 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: b2d2c8b
>2012-12-17 08:34:21,731 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 72cc3a29
>2012-12-17 08:34:21,731 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [28319bba] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:34:24,977 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:34:29,520 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1a2cf793] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:34:29,760 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [7f190dc3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:34:29,837 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 427fa9cf
>2012-12-17 08:34:29,838 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 57c06c82
>2012-12-17 08:34:29,858 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:34:30,799 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SpmStopVDSCommand, log id: 57c06c82
>2012-12-17 08:34:30,848 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, ResetIrsVDSCommand, log id: 427fa9cf
>2012-12-17 08:34:30,859 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 11e573e
>2012-12-17 08:34:30,859 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:34:30,859 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 11e573e
>2012-12-17 08:34:35,210 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:34:41,015 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2c13a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:34:41,057 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2c13a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4583
>2012-12-17 08:34:41,058 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2c13a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4583, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4303267a
>2012-12-17 08:34:41,080 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2c13a] spmStart polling started: taskId = 68d8add5-05f3-43e9-850e-74f29c1d1869
>2012-12-17 08:34:42,102 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2c13a] spmStart polling ended: taskId = 68d8add5-05f3-43e9-850e-74f29c1d1869 task status = finished
>2012-12-17 08:34:42,121 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2c13a] spmStart polling ended, spm status: SPM
>2012-12-17 08:34:42,122 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2c13a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=68d8add5-05f3-43e9-850e-74f29c1d1869), log id: 59ae6494
>2012-12-17 08:34:42,141 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2c13a] FINISH, HSMClearTaskVDSCommand, log id: 59ae6494
>2012-12-17 08:34:42,141 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2c13a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@484d5e8b, log id: 4303267a
>2012-12-17 08:34:42,154 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2c13a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:34:42,202 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [1981e345] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:34:42,202 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3f20e7da
>2012-12-17 08:34:42,230 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2c13a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:34:42,370 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [170be1f2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:34:42,504 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [170be1f2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4f6d2ed
>2012-12-17 08:34:42,506 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [170be1f2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6f16dd59
>2012-12-17 08:34:42,526 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [170be1f2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:34:42,660 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [170be1f2] FINISH, SpmStopVDSCommand, log id: 6f16dd59
>2012-12-17 08:34:42,674 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [170be1f2] FINISH, ResetIrsVDSCommand, log id: 4f6d2ed
>2012-12-17 08:34:42,684 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [170be1f2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4bbc53e3
>2012-12-17 08:34:42,685 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [170be1f2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:34:42,685 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [170be1f2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4bbc53e3
>2012-12-17 08:34:42,738 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1981e345] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:34:42,752 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1981e345] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4584
>2012-12-17 08:34:42,753 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4584, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3a64197d
>2012-12-17 08:34:42,775 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1981e345] spmStart polling started: taskId = f85619b9-5674-41aa-a188-b618b3588afe
>2012-12-17 08:34:43,797 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1981e345] spmStart polling ended: taskId = f85619b9-5674-41aa-a188-b618b3588afe task status = finished
>2012-12-17 08:34:43,815 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1981e345] spmStart polling ended, spm status: SPM
>2012-12-17 08:34:43,816 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f85619b9-5674-41aa-a188-b618b3588afe), log id: 3855735
>2012-12-17 08:34:43,835 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, HSMClearTaskVDSCommand, log id: 3855735
>2012-12-17 08:34:43,835 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@27205a61, log id: 3a64197d
>2012-12-17 08:34:43,851 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1981e345] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:34:43,873 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:34:43,875 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 67f5506
>2012-12-17 08:34:43,893 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 67f5506
>2012-12-17 08:34:43,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f20e7da
>2012-12-17 08:34:43,894 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [1981e345] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:34:43,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 11a7003f
>2012-12-17 08:34:43,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:34:43,895 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6a055145
>2012-12-17 08:34:43,913 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6a055145
>2012-12-17 08:34:43,913 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 11a7003f
>2012-12-17 08:34:43,913 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [1981e345] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:34:46,589 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:34:52,788 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [346cd466] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:34:52,888 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [39b86af9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:34:52,945 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 8a2feb5
>2012-12-17 08:34:52,946 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 525f3d86
>2012-12-17 08:34:52,966 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:34:53,592 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] FINISH, SpmStopVDSCommand, log id: 525f3d86
>2012-12-17 08:34:53,625 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] FINISH, ResetIrsVDSCommand, log id: 8a2feb5
>2012-12-17 08:34:53,636 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: fa306bf
>2012-12-17 08:34:53,636 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [39b86af9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:34:53,636 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: fa306bf
>2012-12-17 08:34:56,806 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:35:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [2edbd860] Checking autorecoverable hosts
>2012-12-17 08:35:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [2edbd860] Autorecovering 0 hosts
>2012-12-17 08:35:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [2edbd860] Checking autorecoverable hosts done
>2012-12-17 08:35:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [2edbd860] Autorecovering storage domains is disabled, skipping
>2012-12-17 08:35:03,697 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [73dddeb9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:35:03,727 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [73dddeb9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4585
>2012-12-17 08:35:03,728 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4585, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 648d3b16
>2012-12-17 08:35:03,750 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] spmStart polling started: taskId = 40d4a8a7-5275-445f-a812-f59d3916fad3
>2012-12-17 08:35:04,772 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] spmStart polling ended: taskId = 40d4a8a7-5275-445f-a812-f59d3916fad3 task status = finished
>2012-12-17 08:35:04,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] spmStart polling ended, spm status: SPM
>2012-12-17 08:35:04,791 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=40d4a8a7-5275-445f-a812-f59d3916fad3), log id: 74b3b0b
>2012-12-17 08:35:04,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] FINISH, HSMClearTaskVDSCommand, log id: 74b3b0b
>2012-12-17 08:35:04,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5c1152ea, log id: 648d3b16
>2012-12-17 08:35:04,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [73dddeb9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:35:04,868 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [419474a0] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:35:04,868 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2a273f21
>2012-12-17 08:35:04,895 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [73dddeb9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:35:05,038 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [39a5c05e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:35:05,084 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7eae072e
>2012-12-17 08:35:05,085 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 78e3a230
>2012-12-17 08:35:05,111 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:05,478 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SpmStopVDSCommand, log id: 78e3a230
>2012-12-17 08:35:06,384 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, ResetIrsVDSCommand, log id: 7eae072e
>2012-12-17 08:35:06,395 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 106387e1
>2012-12-17 08:35:06,395 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:06,395 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 106387e1
>2012-12-17 08:35:06,740 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:35:06,758 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4586
>2012-12-17 08:35:06,760 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4586, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7e37d2f6
>2012-12-17 08:35:06,782 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] spmStart polling started: taskId = 5f4f10b3-7a51-4b4c-8f8c-5e7651d4e327
>2012-12-17 08:35:07,803 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] spmStart polling ended: taskId = 5f4f10b3-7a51-4b4c-8f8c-5e7651d4e327 task status = finished
>2012-12-17 08:35:07,821 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] spmStart polling ended, spm status: SPM
>2012-12-17 08:35:07,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5f4f10b3-7a51-4b4c-8f8c-5e7651d4e327), log id: 75fa42b7
>2012-12-17 08:35:07,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, HSMClearTaskVDSCommand, log id: 75fa42b7
>2012-12-17 08:35:07,841 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@13238e4, log id: 7e37d2f6
>2012-12-17 08:35:07,851 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:35:07,873 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:07,874 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 451a1a97
>2012-12-17 08:35:07,893 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 451a1a97
>2012-12-17 08:35:07,893 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2a273f21
>2012-12-17 08:35:07,894 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [419474a0] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:35:07,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 34c7d2f2
>2012-12-17 08:35:07,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:07,895 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5af96ac9
>2012-12-17 08:35:07,913 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5af96ac9
>2012-12-17 08:35:07,913 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 34c7d2f2
>2012-12-17 08:35:07,913 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [419474a0] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:35:16,788 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [57c34106] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:35:16,908 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [1390e512] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:35:16,957 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [1390e512] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17aa58a1
>2012-12-17 08:35:16,959 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [1390e512] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4182df9a
>2012-12-17 08:35:16,979 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [1390e512] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:17,198 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [780c369f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:35:17,213 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [1390e512] FINISH, SpmStopVDSCommand, log id: 4182df9a
>2012-12-17 08:35:17,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [1390e512] FINISH, ResetIrsVDSCommand, log id: 17aa58a1
>2012-12-17 08:35:17,276 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [1390e512] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 75dcc2e2
>2012-12-17 08:35:17,276 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:17,276 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [1390e512] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 75dcc2e2
>2012-12-17 08:35:27,356 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [8c4c643] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:35:27,387 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [8c4c643] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4587
>2012-12-17 08:35:27,389 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4587, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 530d461f
>2012-12-17 08:35:27,414 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] spmStart polling started: taskId = 988539f1-3971-4845-a874-036ab6a601aa
>2012-12-17 08:35:28,436 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] spmStart polling ended: taskId = 988539f1-3971-4845-a874-036ab6a601aa task status = finished
>2012-12-17 08:35:28,455 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] spmStart polling ended, spm status: SPM
>2012-12-17 08:35:28,456 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=988539f1-3971-4845-a874-036ab6a601aa), log id: 3168740a
>2012-12-17 08:35:28,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, HSMClearTaskVDSCommand, log id: 3168740a
>2012-12-17 08:35:28,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@409a55dd, log id: 530d461f
>2012-12-17 08:35:28,505 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [8c4c643] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:35:28,556 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [2f70811] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:35:28,556 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3f9ca247
>2012-12-17 08:35:28,574 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [8c4c643] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:35:28,710 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [653dcf69] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:35:28,757 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 549e38a2
>2012-12-17 08:35:28,758 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 762956ff
>2012-12-17 08:35:28,779 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:28,905 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] FINISH, SpmStopVDSCommand, log id: 762956ff
>2012-12-17 08:35:28,921 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] FINISH, ResetIrsVDSCommand, log id: 549e38a2
>2012-12-17 08:35:28,932 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6f7aba17
>2012-12-17 08:35:28,932 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [653dcf69] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:28,932 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6f7aba17
>2012-12-17 08:35:28,981 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:35:28,995 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4588
>2012-12-17 08:35:28,998 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4588, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: c629138
>2012-12-17 08:35:29,021 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] spmStart polling started: taskId = 65e66282-cb57-486b-93a2-e2587a521e6f
>2012-12-17 08:35:30,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] spmStart polling ended: taskId = 65e66282-cb57-486b-93a2-e2587a521e6f task status = finished
>2012-12-17 08:35:30,060 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] spmStart polling ended, spm status: SPM
>2012-12-17 08:35:30,061 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=65e66282-cb57-486b-93a2-e2587a521e6f), log id: 1c59b6a3
>2012-12-17 08:35:30,080 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, HSMClearTaskVDSCommand, log id: 1c59b6a3
>2012-12-17 08:35:30,080 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@45e595df, log id: c629138
>2012-12-17 08:35:30,096 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:35:30,108 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:30,113 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5573ac6e
>2012-12-17 08:35:30,132 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5573ac6e
>2012-12-17 08:35:30,132 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f9ca247
>2012-12-17 08:35:30,132 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [2f70811] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:35:30,132 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3627b9d9
>2012-12-17 08:35:30,132 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:30,134 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 463d929c
>2012-12-17 08:35:30,153 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 463d929c
>2012-12-17 08:35:30,153 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3627b9d9
>2012-12-17 08:35:30,153 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [2f70811] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:35:37,602 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4579c6a9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:35:39,033 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [323f195a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:35:39,309 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:35:39,362 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 52fecd18
>2012-12-17 08:35:39,363 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5272d96a
>2012-12-17 08:35:39,383 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:39,589 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] FINISH, SpmStopVDSCommand, log id: 5272d96a
>2012-12-17 08:35:39,616 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] FINISH, ResetIrsVDSCommand, log id: 52fecd18
>2012-12-17 08:35:39,639 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 56e4aea2
>2012-12-17 08:35:39,639 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:39,639 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 56e4aea2
>2012-12-17 08:35:47,857 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:35:49,736 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69e82ff1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:35:49,754 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69e82ff1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4589
>2012-12-17 08:35:49,755 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4589, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2569beda
>2012-12-17 08:35:49,777 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] spmStart polling started: taskId = ebea22bc-19b0-48be-a460-f5d9da4b2d04
>2012-12-17 08:35:50,801 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] spmStart polling ended: taskId = ebea22bc-19b0-48be-a460-f5d9da4b2d04 task status = finished
>2012-12-17 08:35:50,820 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] spmStart polling ended, spm status: SPM
>2012-12-17 08:35:50,821 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ebea22bc-19b0-48be-a460-f5d9da4b2d04), log id: 164a5143
>2012-12-17 08:35:50,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] FINISH, HSMClearTaskVDSCommand, log id: 164a5143
>2012-12-17 08:35:50,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@691d1123, log id: 2569beda
>2012-12-17 08:35:50,981 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69e82ff1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:35:51,426 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [59932b3] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:35:51,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 12b28511
>2012-12-17 08:35:51,593 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69e82ff1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:35:51,721 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [369f35cc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:35:51,791 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 44925485
>2012-12-17 08:35:51,793 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 446cf875
>2012-12-17 08:35:51,814 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:51,951 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] FINISH, SpmStopVDSCommand, log id: 446cf875
>2012-12-17 08:35:51,962 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] FINISH, ResetIrsVDSCommand, log id: 44925485
>2012-12-17 08:35:51,974 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 9400c78
>2012-12-17 08:35:51,974 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [369f35cc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:51,974 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 9400c78
>2012-12-17 08:35:52,031 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:35:52,059 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4590
>2012-12-17 08:35:52,061 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4590, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62d399eb
>2012-12-17 08:35:52,084 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] spmStart polling started: taskId = 3f13cba0-32a4-4a35-abb1-9f606bb44bc2
>2012-12-17 08:35:53,107 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] spmStart polling ended: taskId = 3f13cba0-32a4-4a35-abb1-9f606bb44bc2 task status = finished
>2012-12-17 08:35:53,126 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] spmStart polling ended, spm status: SPM
>2012-12-17 08:35:53,127 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3f13cba0-32a4-4a35-abb1-9f606bb44bc2), log id: 11628e8e
>2012-12-17 08:35:53,145 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, HSMClearTaskVDSCommand, log id: 11628e8e
>2012-12-17 08:35:53,145 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d7d158c, log id: 62d399eb
>2012-12-17 08:35:53,155 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:35:53,165 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:53,169 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5b301a20
>2012-12-17 08:35:53,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5b301a20
>2012-12-17 08:35:53,189 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 12b28511
>2012-12-17 08:35:53,189 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [59932b3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:35:53,189 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 20f279cd
>2012-12-17 08:35:53,189 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:35:53,191 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 72c887b
>2012-12-17 08:35:53,209 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 72c887b
>2012-12-17 08:35:53,209 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 20f279cd
>2012-12-17 08:35:53,209 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [59932b3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:35:58,036 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7f13ba47] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:36:02,080 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [65976fd0] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:36:02,228 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [639cabd2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:36:02,275 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 118c427f
>2012-12-17 08:36:02,277 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1d05b855
>2012-12-17 08:36:02,297 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:02,444 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] FINISH, SpmStopVDSCommand, log id: 1d05b855
>2012-12-17 08:36:02,465 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] FINISH, ResetIrsVDSCommand, log id: 118c427f
>2012-12-17 08:36:02,476 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4a3da75e
>2012-12-17 08:36:02,476 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [639cabd2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:02,476 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4a3da75e
>2012-12-17 08:36:08,241 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [95f5063] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:36:12,530 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [70373480] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:36:12,549 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [70373480] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4591
>2012-12-17 08:36:12,551 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [70373480] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4591, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 47fd183b
>2012-12-17 08:36:12,573 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [70373480] spmStart polling started: taskId = 57f689e4-44ea-40d3-865a-a577d1377028
>2012-12-17 08:36:13,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [70373480] spmStart polling ended: taskId = 57f689e4-44ea-40d3-865a-a577d1377028 task status = finished
>2012-12-17 08:36:13,614 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [70373480] spmStart polling ended, spm status: SPM
>2012-12-17 08:36:13,615 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [70373480] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=57f689e4-44ea-40d3-865a-a577d1377028), log id: 2472cb60
>2012-12-17 08:36:13,634 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [70373480] FINISH, HSMClearTaskVDSCommand, log id: 2472cb60
>2012-12-17 08:36:13,634 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [70373480] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6c861e51, log id: 47fd183b
>2012-12-17 08:36:13,650 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [70373480] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:36:13,698 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:36:13,698 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6f12c5d1
>2012-12-17 08:36:13,741 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [70373480] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:36:13,873 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [1daf49b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:36:13,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 13da28bf
>2012-12-17 08:36:13,923 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 25f5444a
>2012-12-17 08:36:13,942 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:14,070 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] FINISH, SpmStopVDSCommand, log id: 25f5444a
>2012-12-17 08:36:14,083 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] FINISH, ResetIrsVDSCommand, log id: 13da28bf
>2012-12-17 08:36:14,094 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7c78c5ef
>2012-12-17 08:36:14,094 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1daf49b5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:14,094 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7c78c5ef
>2012-12-17 08:36:14,160 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:36:14,184 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4592
>2012-12-17 08:36:14,185 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4592, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 26effec0
>2012-12-17 08:36:14,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling started: taskId = b24585df-ea97-4806-bbdb-31e52e15122d
>2012-12-17 08:36:15,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling ended: taskId = b24585df-ea97-4806-bbdb-31e52e15122d task status = finished
>2012-12-17 08:36:15,246 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling ended, spm status: SPM
>2012-12-17 08:36:15,248 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b24585df-ea97-4806-bbdb-31e52e15122d), log id: 260545d4
>2012-12-17 08:36:15,266 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMClearTaskVDSCommand, log id: 260545d4
>2012-12-17 08:36:15,266 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@242b74cc, log id: 26effec0
>2012-12-17 08:36:15,280 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:36:15,304 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:15,309 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 422df258
>2012-12-17 08:36:15,329 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 422df258
>2012-12-17 08:36:15,329 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6f12c5d1
>2012-12-17 08:36:15,329 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:36:15,329 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 546d966d
>2012-12-17 08:36:15,329 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:15,330 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4cc0d581
>2012-12-17 08:36:15,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4cc0d581
>2012-12-17 08:36:15,349 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 546d966d
>2012-12-17 08:36:15,349 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:36:18,504 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:36:24,223 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4579c6a9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:36:24,361 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [4f93d023] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:36:24,415 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 57d3e7c5
>2012-12-17 08:36:24,416 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1ec3e2a8
>2012-12-17 08:36:24,435 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:24,584 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] FINISH, SpmStopVDSCommand, log id: 1ec3e2a8
>2012-12-17 08:36:24,608 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] FINISH, ResetIrsVDSCommand, log id: 57d3e7c5
>2012-12-17 08:36:24,622 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2ff10101
>2012-12-17 08:36:24,623 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4f93d023] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:24,623 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2ff10101
>2012-12-17 08:36:28,700 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:36:34,797 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [69d0a8ae] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:36:34,822 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [69d0a8ae] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4593
>2012-12-17 08:36:34,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4593, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 52d1342c
>2012-12-17 08:36:34,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] spmStart polling started: taskId = 1c819230-12f8-4001-83f1-0a3a4a0c53bd
>2012-12-17 08:36:35,869 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] spmStart polling ended: taskId = 1c819230-12f8-4001-83f1-0a3a4a0c53bd task status = finished
>2012-12-17 08:36:35,887 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] spmStart polling ended, spm status: SPM
>2012-12-17 08:36:35,888 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1c819230-12f8-4001-83f1-0a3a4a0c53bd), log id: 21aba7e5
>2012-12-17 08:36:35,906 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] FINISH, HSMClearTaskVDSCommand, log id: 21aba7e5
>2012-12-17 08:36:35,906 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@c9bbca8, log id: 52d1342c
>2012-12-17 08:36:36,141 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [69d0a8ae] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:36:36,261 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [2edbd860] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:36:36,261 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5b380e7b
>2012-12-17 08:36:36,283 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [69d0a8ae] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:36:36,449 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [24452d4e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:36:36,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 427cea7e
>2012-12-17 08:36:36,544 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26c9613b
>2012-12-17 08:36:36,563 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:36,684 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] FINISH, SpmStopVDSCommand, log id: 26c9613b
>2012-12-17 08:36:36,694 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] FINISH, ResetIrsVDSCommand, log id: 427cea7e
>2012-12-17 08:36:36,705 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 239678d0
>2012-12-17 08:36:36,705 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [24452d4e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:36,705 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 239678d0
>2012-12-17 08:36:36,756 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2edbd860] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:36:36,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2edbd860] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4594
>2012-12-17 08:36:36,767 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4594, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 23f375e8
>2012-12-17 08:36:36,788 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] spmStart polling started: taskId = 1a78f8e1-e9c1-4195-8414-aeebec9a07ed
>2012-12-17 08:36:37,813 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] spmStart polling ended: taskId = 1a78f8e1-e9c1-4195-8414-aeebec9a07ed task status = finished
>2012-12-17 08:36:37,831 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] spmStart polling ended, spm status: SPM
>2012-12-17 08:36:37,832 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1a78f8e1-e9c1-4195-8414-aeebec9a07ed), log id: 73bf7cbf
>2012-12-17 08:36:37,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, HSMClearTaskVDSCommand, log id: 73bf7cbf
>2012-12-17 08:36:37,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@20971af7, log id: 23f375e8
>2012-12-17 08:36:37,864 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2edbd860] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:36:37,898 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:37,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3ec55324
>2012-12-17 08:36:37,919 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3ec55324
>2012-12-17 08:36:37,919 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5b380e7b
>2012-12-17 08:36:37,919 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [2edbd860] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:36:37,920 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 14f32617
>2012-12-17 08:36:37,920 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:37,921 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5985a781
>2012-12-17 08:36:37,940 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5985a781
>2012-12-17 08:36:37,940 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 14f32617
>2012-12-17 08:36:37,940 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [2edbd860] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:36:38,948 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [717cca29] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:36:46,796 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [676a97fd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:36:46,949 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-75) [4232612d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:36:46,992 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [4232612d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3904c3d3
>2012-12-17 08:36:46,993 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [4232612d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 51e8c35c
>2012-12-17 08:36:47,013 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [4232612d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:47,266 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [4232612d] FINISH, SpmStopVDSCommand, log id: 51e8c35c
>2012-12-17 08:36:47,277 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [4232612d] FINISH, ResetIrsVDSCommand, log id: 3904c3d3
>2012-12-17 08:36:47,288 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [4232612d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5eacdb29
>2012-12-17 08:36:47,288 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4232612d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:47,288 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [4232612d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5eacdb29
>2012-12-17 08:36:49,159 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:36:57,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:36:57,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4595
>2012-12-17 08:36:57,439 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4595, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: a8f17c1
>2012-12-17 08:36:57,462 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] spmStart polling started: taskId = 8fd3289e-496c-4669-b7b8-6827775c6d66
>2012-12-17 08:36:58,484 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] spmStart polling ended: taskId = 8fd3289e-496c-4669-b7b8-6827775c6d66 task status = finished
>2012-12-17 08:36:58,502 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] spmStart polling ended, spm status: SPM
>2012-12-17 08:36:58,504 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8fd3289e-496c-4669-b7b8-6827775c6d66), log id: 71f2d1a7
>2012-12-17 08:36:58,522 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, HSMClearTaskVDSCommand, log id: 71f2d1a7
>2012-12-17 08:36:58,522 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7ca26562, log id: a8f17c1
>2012-12-17 08:36:58,535 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:36:58,632 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [169ff372] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:36:58,632 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6832f6d
>2012-12-17 08:36:58,652 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:36:58,775 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-18) [1ee6f70] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:36:58,846 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [1ee6f70] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 452b64e2
>2012-12-17 08:36:58,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [1ee6f70] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 224076e4
>2012-12-17 08:36:58,868 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [1ee6f70] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:58,990 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [1ee6f70] FINISH, SpmStopVDSCommand, log id: 224076e4
>2012-12-17 08:36:59,012 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [1ee6f70] FINISH, ResetIrsVDSCommand, log id: 452b64e2
>2012-12-17 08:36:59,023 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [1ee6f70] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e9f1412
>2012-12-17 08:36:59,023 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [1ee6f70] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:36:59,023 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [1ee6f70] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e9f1412
>2012-12-17 08:36:59,086 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:36:59,105 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4596
>2012-12-17 08:36:59,107 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4596, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 191bfd63
>2012-12-17 08:36:59,128 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] spmStart polling started: taskId = bc21521f-f82b-4e40-9a8d-c569f9e8e2e5
>2012-12-17 08:37:00,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] spmStart polling ended: taskId = bc21521f-f82b-4e40-9a8d-c569f9e8e2e5 task status = finished
>2012-12-17 08:37:00,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] spmStart polling ended, spm status: SPM
>2012-12-17 08:37:00,169 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bc21521f-f82b-4e40-9a8d-c569f9e8e2e5), log id: 172789fc
>2012-12-17 08:37:00,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, HSMClearTaskVDSCommand, log id: 172789fc
>2012-12-17 08:37:00,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@24d40b44, log id: 191bfd63
>2012-12-17 08:37:00,205 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:37:00,218 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:00,220 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3d87ddf0
>2012-12-17 08:37:00,239 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3d87ddf0
>2012-12-17 08:37:00,239 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6832f6d
>2012-12-17 08:37:00,239 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [169ff372] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:37:00,239 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 22fbd71f
>2012-12-17 08:37:00,239 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:00,240 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 40536ce0
>2012-12-17 08:37:00,259 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 40536ce0
>2012-12-17 08:37:00,259 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 22fbd71f
>2012-12-17 08:37:00,259 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [169ff372] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:37:09,137 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [27ff9aa4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:37:09,274 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:37:09,338 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 67920a70
>2012-12-17 08:37:09,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5c4905a7
>2012-12-17 08:37:09,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:09,553 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2f94bcb4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:37:09,613 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, SpmStopVDSCommand, log id: 5c4905a7
>2012-12-17 08:37:09,640 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, ResetIrsVDSCommand, log id: 67920a70
>2012-12-17 08:37:09,651 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 73f4aaea
>2012-12-17 08:37:09,651 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:09,651 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 73f4aaea
>2012-12-17 08:37:19,704 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [733ac637] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:37:19,720 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [733ac637] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4597
>2012-12-17 08:37:19,722 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [733ac637] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4597, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4f67cac3
>2012-12-17 08:37:19,747 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [733ac637] spmStart polling started: taskId = 27dba14f-d616-43e6-bb20-4eb68e6de390
>2012-12-17 08:37:20,771 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [733ac637] spmStart polling ended: taskId = 27dba14f-d616-43e6-bb20-4eb68e6de390 task status = finished
>2012-12-17 08:37:20,789 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [733ac637] spmStart polling ended, spm status: SPM
>2012-12-17 08:37:20,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [733ac637] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=27dba14f-d616-43e6-bb20-4eb68e6de390), log id: 2e1493bf
>2012-12-17 08:37:20,808 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [733ac637] FINISH, HSMClearTaskVDSCommand, log id: 2e1493bf
>2012-12-17 08:37:20,808 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [733ac637] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48eff83b, log id: 4f67cac3
>2012-12-17 08:37:20,985 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [733ac637] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:37:21,142 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:37:21,143 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5194d37f
>2012-12-17 08:37:21,181 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [733ac637] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:37:21,361 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-44) [44f1dfe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:37:21,433 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6910814d
>2012-12-17 08:37:21,435 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2dcfd2cb
>2012-12-17 08:37:21,454 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:21,593 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] FINISH, SpmStopVDSCommand, log id: 2dcfd2cb
>2012-12-17 08:37:21,608 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] FINISH, ResetIrsVDSCommand, log id: 6910814d
>2012-12-17 08:37:21,618 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 12e6c080
>2012-12-17 08:37:21,618 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [44f1dfe] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:21,618 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 12e6c080
>2012-12-17 08:37:21,670 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:37:21,683 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4598
>2012-12-17 08:37:21,684 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4598, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7e284867
>2012-12-17 08:37:21,706 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling started: taskId = ad1d8e50-95da-47f0-a115-f593652f37ec
>2012-12-17 08:37:22,728 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended: taskId = ad1d8e50-95da-47f0-a115-f593652f37ec task status = finished
>2012-12-17 08:37:22,747 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended, spm status: SPM
>2012-12-17 08:37:22,748 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ad1d8e50-95da-47f0-a115-f593652f37ec), log id: 58033b29
>2012-12-17 08:37:22,767 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMClearTaskVDSCommand, log id: 58033b29
>2012-12-17 08:37:22,767 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3c212de3, log id: 7e284867
>2012-12-17 08:37:22,773 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:37:22,794 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:22,796 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7d7605a5
>2012-12-17 08:37:22,815 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7d7605a5
>2012-12-17 08:37:22,815 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5194d37f
>2012-12-17 08:37:22,816 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:37:22,816 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4f6cd3fc
>2012-12-17 08:37:22,816 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:22,817 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 52d6441a
>2012-12-17 08:37:22,835 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 52d6441a
>2012-12-17 08:37:22,835 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4f6cd3fc
>2012-12-17 08:37:22,835 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:37:29,930 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [2afeec77] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:37:31,721 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [47f0782e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:37:31,803 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [50db5e16] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:37:31,907 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 79787e70
>2012-12-17 08:37:31,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5307046d
>2012-12-17 08:37:31,929 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:32,684 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] FINISH, SpmStopVDSCommand, log id: 5307046d
>2012-12-17 08:37:32,695 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] FINISH, ResetIrsVDSCommand, log id: 79787e70
>2012-12-17 08:37:32,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1cb1ca87
>2012-12-17 08:37:32,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [50db5e16] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:32,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1cb1ca87
>2012-12-17 08:37:40,145 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:37:42,756 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [318b1f4f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:37:42,777 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [318b1f4f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4599
>2012-12-17 08:37:42,778 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [318b1f4f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4599, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4c1d3d63
>2012-12-17 08:37:42,802 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [318b1f4f] spmStart polling started: taskId = d7641631-09d0-476d-bcf0-4712138f71fb
>2012-12-17 08:37:43,825 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [318b1f4f] spmStart polling ended: taskId = d7641631-09d0-476d-bcf0-4712138f71fb task status = finished
>2012-12-17 08:37:43,844 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [318b1f4f] spmStart polling ended, spm status: SPM
>2012-12-17 08:37:43,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [318b1f4f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d7641631-09d0-476d-bcf0-4712138f71fb), log id: 41187a7b
>2012-12-17 08:37:43,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [318b1f4f] FINISH, HSMClearTaskVDSCommand, log id: 41187a7b
>2012-12-17 08:37:43,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [318b1f4f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2e1142a0, log id: 4c1d3d63
>2012-12-17 08:37:43,876 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [318b1f4f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:37:43,915 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [cdacbd1] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:37:43,915 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1456527c
>2012-12-17 08:37:43,943 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [318b1f4f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:37:44,076 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-89) [4beddd99] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:37:44,142 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [4beddd99] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 71a07f19
>2012-12-17 08:37:44,144 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [4beddd99] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4115c712
>2012-12-17 08:37:44,164 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [4beddd99] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:44,365 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [4beddd99] FINISH, SpmStopVDSCommand, log id: 4115c712
>2012-12-17 08:37:44,379 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [4beddd99] FINISH, ResetIrsVDSCommand, log id: 71a07f19
>2012-12-17 08:37:44,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [4beddd99] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7a011164
>2012-12-17 08:37:44,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4beddd99] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:44,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [4beddd99] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7a011164
>2012-12-17 08:37:44,443 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:37:44,456 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4600
>2012-12-17 08:37:44,458 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4600, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 43d0f323
>2012-12-17 08:37:44,481 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] spmStart polling started: taskId = 7374690a-afeb-4cf1-93cc-3f5fe1d6c5f0
>2012-12-17 08:37:45,504 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] spmStart polling ended: taskId = 7374690a-afeb-4cf1-93cc-3f5fe1d6c5f0 task status = finished
>2012-12-17 08:37:45,523 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] spmStart polling ended, spm status: SPM
>2012-12-17 08:37:45,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7374690a-afeb-4cf1-93cc-3f5fe1d6c5f0), log id: 516ec621
>2012-12-17 08:37:45,542 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, HSMClearTaskVDSCommand, log id: 516ec621
>2012-12-17 08:37:45,543 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3f91ca4b, log id: 43d0f323
>2012-12-17 08:37:45,725 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:37:46,048 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:46,050 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 220c6a2
>2012-12-17 08:37:46,072 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 220c6a2
>2012-12-17 08:37:46,072 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1456527c
>2012-12-17 08:37:46,072 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [cdacbd1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:37:46,073 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 12a6da07
>2012-12-17 08:37:46,073 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:46,074 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: c43cadd
>2012-12-17 08:37:46,093 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: c43cadd
>2012-12-17 08:37:46,093 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 12a6da07
>2012-12-17 08:37:46,093 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [cdacbd1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:37:50,966 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [653dcf69] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:37:54,481 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1981e345] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:37:54,650 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [5266d504] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:37:54,697 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [5266d504] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 15389550
>2012-12-17 08:37:54,698 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [5266d504] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e04a16
>2012-12-17 08:37:54,718 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [5266d504] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:55,484 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [5266d504] FINISH, SpmStopVDSCommand, log id: 7e04a16
>2012-12-17 08:37:55,844 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [5266d504] FINISH, ResetIrsVDSCommand, log id: 15389550
>2012-12-17 08:37:55,857 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [5266d504] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40787858
>2012-12-17 08:37:55,857 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5266d504] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:37:55,857 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [5266d504] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40787858
>2012-12-17 08:38:01,259 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [58c21a2d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:38:06,452 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:38:06,500 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4601
>2012-12-17 08:38:06,501 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4601, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1e50f11c
>2012-12-17 08:38:06,525 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] spmStart polling started: taskId = 14227764-0b58-4f8b-ad3f-09edb98a1c8d
>2012-12-17 08:38:07,546 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] spmStart polling ended: taskId = 14227764-0b58-4f8b-ad3f-09edb98a1c8d task status = finished
>2012-12-17 08:38:07,564 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] spmStart polling ended, spm status: SPM
>2012-12-17 08:38:07,565 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=14227764-0b58-4f8b-ad3f-09edb98a1c8d), log id: 6272c868
>2012-12-17 08:38:07,583 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, HSMClearTaskVDSCommand, log id: 6272c868
>2012-12-17 08:38:07,583 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5dc35656, log id: 1e50f11c
>2012-12-17 08:38:07,597 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:38:07,640 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:38:07,640 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2aaf26a9
>2012-12-17 08:38:07,674 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:38:07,880 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [56986c52] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:38:07,979 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [56986c52] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1cf53cdf
>2012-12-17 08:38:07,980 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [56986c52] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4cfea686
>2012-12-17 08:38:08,000 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [56986c52] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:08,141 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [56986c52] FINISH, SpmStopVDSCommand, log id: 4cfea686
>2012-12-17 08:38:08,215 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [56986c52] FINISH, ResetIrsVDSCommand, log id: 1cf53cdf
>2012-12-17 08:38:08,226 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [56986c52] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 734c2cf5
>2012-12-17 08:38:08,226 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [56986c52] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:08,226 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [56986c52] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 734c2cf5
>2012-12-17 08:38:08,297 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:38:08,308 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4602
>2012-12-17 08:38:08,309 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4602, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1d52348e
>2012-12-17 08:38:08,331 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling started: taskId = ea11719f-75d5-4c8a-88b7-05492448ddbf
>2012-12-17 08:38:09,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended: taskId = ea11719f-75d5-4c8a-88b7-05492448ddbf task status = finished
>2012-12-17 08:38:09,371 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended, spm status: SPM
>2012-12-17 08:38:09,372 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ea11719f-75d5-4c8a-88b7-05492448ddbf), log id: dcf542d
>2012-12-17 08:38:09,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMClearTaskVDSCommand, log id: dcf542d
>2012-12-17 08:38:09,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2a6b50e2, log id: 1d52348e
>2012-12-17 08:38:09,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:38:09,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:09,428 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 52c8436e
>2012-12-17 08:38:09,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 52c8436e
>2012-12-17 08:38:09,449 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2aaf26a9
>2012-12-17 08:38:09,449 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:38:09,449 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 16a5fbeb
>2012-12-17 08:38:09,449 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:09,451 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2aaa7331
>2012-12-17 08:38:09,469 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2aaa7331
>2012-12-17 08:38:09,469 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 16a5fbeb
>2012-12-17 08:38:09,469 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:38:11,553 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [5a98cdcf] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:38:18,389 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [2afeec77] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:38:18,542 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [1f4324c3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:38:18,602 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: b062fbd
>2012-12-17 08:38:18,603 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2e542f3b
>2012-12-17 08:38:18,623 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:18,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, SpmStopVDSCommand, log id: 2e542f3b
>2012-12-17 08:38:18,819 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, ResetIrsVDSCommand, log id: b062fbd
>2012-12-17 08:38:18,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 28a27b45
>2012-12-17 08:38:18,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:18,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 28a27b45
>2012-12-17 08:38:21,754 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [495d106] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:38:28,880 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [6e64c0c8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:38:28,907 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [6e64c0c8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4603
>2012-12-17 08:38:28,909 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [6e64c0c8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4603, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6326ca34
>2012-12-17 08:38:28,932 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [6e64c0c8] spmStart polling started: taskId = a80b6299-28e2-43e9-a6e3-0d18c0814ad8
>2012-12-17 08:38:29,953 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [6e64c0c8] spmStart polling ended: taskId = a80b6299-28e2-43e9-a6e3-0d18c0814ad8 task status = finished
>2012-12-17 08:38:29,971 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [6e64c0c8] spmStart polling ended, spm status: SPM
>2012-12-17 08:38:29,972 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [6e64c0c8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a80b6299-28e2-43e9-a6e3-0d18c0814ad8), log id: 64a4d108
>2012-12-17 08:38:29,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [6e64c0c8] FINISH, HSMClearTaskVDSCommand, log id: 64a4d108
>2012-12-17 08:38:29,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [6e64c0c8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@9797d7, log id: 6326ca34
>2012-12-17 08:38:30,012 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [6e64c0c8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:38:30,048 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [95f5063] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:38:30,048 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 578fe9e8
>2012-12-17 08:38:30,077 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [6e64c0c8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:38:30,205 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-99) [b3b23b2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:38:30,262 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 79ae25f7
>2012-12-17 08:38:30,264 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5ed09b6a
>2012-12-17 08:38:30,283 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:30,843 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, SpmStopVDSCommand, log id: 5ed09b6a
>2012-12-17 08:38:30,926 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, ResetIrsVDSCommand, log id: 79ae25f7
>2012-12-17 08:38:30,937 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 19cf0981
>2012-12-17 08:38:30,937 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:30,937 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 19cf0981
>2012-12-17 08:38:31,198 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [95f5063] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:38:31,361 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [95f5063] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4604
>2012-12-17 08:38:31,363 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4604, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5d81057a
>2012-12-17 08:38:31,385 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [95f5063] spmStart polling started: taskId = 0bc9cc0d-4c7b-4371-89da-b28ea9b5a412
>2012-12-17 08:38:32,406 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [95f5063] spmStart polling ended: taskId = 0bc9cc0d-4c7b-4371-89da-b28ea9b5a412 task status = finished
>2012-12-17 08:38:32,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [95f5063] spmStart polling ended, spm status: SPM
>2012-12-17 08:38:32,425 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0bc9cc0d-4c7b-4371-89da-b28ea9b5a412), log id: 89689e0
>2012-12-17 08:38:32,444 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, HSMClearTaskVDSCommand, log id: 89689e0
>2012-12-17 08:38:32,444 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5335d587, log id: 5d81057a
>2012-12-17 08:38:32,467 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [95f5063] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:38:32,483 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:32,487 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 190546fe
>2012-12-17 08:38:32,511 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 190546fe
>2012-12-17 08:38:32,511 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 578fe9e8
>2012-12-17 08:38:32,511 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [95f5063] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:38:32,511 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2942d4b0
>2012-12-17 08:38:32,511 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:32,513 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6a7cddae
>2012-12-17 08:38:32,542 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6a7cddae
>2012-12-17 08:38:32,542 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2942d4b0
>2012-12-17 08:38:32,542 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [95f5063] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:38:41,584 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:38:41,750 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [41698d74] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:38:41,845 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 284bfdb3
>2012-12-17 08:38:41,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 406d69c7
>2012-12-17 08:38:41,866 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [41698d74] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:42,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, SpmStopVDSCommand, log id: 406d69c7
>2012-12-17 08:38:42,055 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, ResetIrsVDSCommand, log id: 284bfdb3
>2012-12-17 08:38:42,066 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a9bd265
>2012-12-17 08:38:42,066 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [41698d74] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:42,066 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a9bd265
>2012-12-17 08:38:42,164 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:38:52,132 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [39b86af9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:38:52,278 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [39b86af9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4605
>2012-12-17 08:38:52,280 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4605, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 23022fd2
>2012-12-17 08:38:52,322 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] spmStart polling started: taskId = 449b3ba0-4929-4634-9128-0a425c3f5874
>2012-12-17 08:38:53,344 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] spmStart polling ended: taskId = 449b3ba0-4929-4634-9128-0a425c3f5874 task status = finished
>2012-12-17 08:38:53,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] spmStart polling ended, spm status: SPM
>2012-12-17 08:38:53,364 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=449b3ba0-4929-4634-9128-0a425c3f5874), log id: 6ef11e6d
>2012-12-17 08:38:53,382 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] FINISH, HSMClearTaskVDSCommand, log id: 6ef11e6d
>2012-12-17 08:38:53,382 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1804bf02, log id: 23022fd2
>2012-12-17 08:38:53,393 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [39b86af9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:38:53,425 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [406132e6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:38:53,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4136adc8
>2012-12-17 08:38:53,461 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [39b86af9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:38:53,545 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [25db9572] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:38:53,595 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [25db9572] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 724b53
>2012-12-17 08:38:53,596 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [25db9572] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 57a8d71
>2012-12-17 08:38:53,615 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [25db9572] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:53,758 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [25db9572] FINISH, SpmStopVDSCommand, log id: 57a8d71
>2012-12-17 08:38:53,770 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [25db9572] FINISH, ResetIrsVDSCommand, log id: 724b53
>2012-12-17 08:38:53,781 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [25db9572] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49c8b150
>2012-12-17 08:38:53,781 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [25db9572] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:53,781 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [25db9572] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49c8b150
>2012-12-17 08:38:53,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:38:53,845 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4606
>2012-12-17 08:38:53,846 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4606, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7164aa23
>2012-12-17 08:38:53,869 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] spmStart polling started: taskId = e9704d9e-eeed-4a50-a97c-07c08f8f2325
>2012-12-17 08:38:54,890 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] spmStart polling ended: taskId = e9704d9e-eeed-4a50-a97c-07c08f8f2325 task status = finished
>2012-12-17 08:38:54,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] spmStart polling ended, spm status: SPM
>2012-12-17 08:38:54,910 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e9704d9e-eeed-4a50-a97c-07c08f8f2325), log id: 55598f7a
>2012-12-17 08:38:54,927 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, HSMClearTaskVDSCommand, log id: 55598f7a
>2012-12-17 08:38:54,927 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5dd65fa2, log id: 7164aa23
>2012-12-17 08:38:54,942 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:38:54,984 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:54,986 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: b7cfaef
>2012-12-17 08:38:55,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: b7cfaef
>2012-12-17 08:38:55,007 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4136adc8
>2012-12-17 08:38:55,007 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [406132e6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:38:55,007 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3f1080fc
>2012-12-17 08:38:55,007 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:38:55,009 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 767259a7
>2012-12-17 08:38:55,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 767259a7
>2012-12-17 08:38:55,027 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f1080fc
>2012-12-17 08:38:55,027 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [406132e6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:39:02,629 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [717cca29] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:39:03,872 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:39:04,027 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-56) [1a6bc63] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:39:04,119 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [1a6bc63] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4a2471d7
>2012-12-17 08:39:04,121 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [1a6bc63] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 34aff74e
>2012-12-17 08:39:04,141 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [1a6bc63] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:39:04,681 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [1a6bc63] FINISH, SpmStopVDSCommand, log id: 34aff74e
>2012-12-17 08:39:04,688 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [1a6bc63] FINISH, ResetIrsVDSCommand, log id: 4a2471d7
>2012-12-17 08:39:04,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [1a6bc63] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 750565d7
>2012-12-17 08:39:04,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [1a6bc63] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:39:04,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [1a6bc63] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 750565d7
>2012-12-17 08:39:12,868 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4232612d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:39:14,752 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [639cabd2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:39:14,769 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [639cabd2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4607
>2012-12-17 08:39:14,770 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4607, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 20dcbe8d
>2012-12-17 08:39:14,791 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] spmStart polling started: taskId = 51c7c47e-e18e-4f82-ab0d-06f9a9d592d4
>2012-12-17 08:39:15,813 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] spmStart polling ended: taskId = 51c7c47e-e18e-4f82-ab0d-06f9a9d592d4 task status = finished
>2012-12-17 08:39:15,831 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] spmStart polling ended, spm status: SPM
>2012-12-17 08:39:15,832 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=51c7c47e-e18e-4f82-ab0d-06f9a9d592d4), log id: 42c79c9e
>2012-12-17 08:39:15,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] FINISH, HSMClearTaskVDSCommand, log id: 42c79c9e
>2012-12-17 08:39:15,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3efb6b52, log id: 20dcbe8d
>2012-12-17 08:39:16,062 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [639cabd2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:39:16,114 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-17) [a2fcda8] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:39:16,114 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 20826965
>2012-12-17 08:39:16,140 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [639cabd2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:39:16,246 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [7d3891a6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:39:16,297 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2d7fc4d9
>2012-12-17 08:39:16,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3c7ef659
>2012-12-17 08:39:16,319 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:39:16,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, SpmStopVDSCommand, log id: 3c7ef659
>2012-12-17 08:39:16,796 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, ResetIrsVDSCommand, log id: 2d7fc4d9
>2012-12-17 08:39:16,807 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3200bb49
>2012-12-17 08:39:16,807 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7d3891a6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:39:16,807 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3200bb49
>2012-12-17 08:39:16,897 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [a2fcda8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:39:16,914 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [a2fcda8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4608
>2012-12-17 08:39:16,916 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4608, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2bc778bc
>2012-12-17 08:39:16,943 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] spmStart polling started: taskId = a10956fa-88f8-4ede-9e78-33268e3283de
>2012-12-17 08:39:17,967 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] spmStart polling ended: taskId = a10956fa-88f8-4ede-9e78-33268e3283de task status = finished
>2012-12-17 08:39:17,986 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] spmStart polling ended, spm status: SPM
>2012-12-17 08:39:17,988 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a10956fa-88f8-4ede-9e78-33268e3283de), log id: 7936bf4c
>2012-12-17 08:39:18,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] FINISH, HSMClearTaskVDSCommand, log id: 7936bf4c
>2012-12-17 08:39:18,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@794954bf, log id: 2bc778bc
>2012-12-17 08:39:18,016 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [a2fcda8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:39:18,052 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:39:18,056 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1545f3f0
>2012-12-17 08:39:18,076 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1545f3f0
>2012-12-17 08:39:18,076 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 20826965
>2012-12-17 08:39:18,076 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-17) [a2fcda8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:39:18,077 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 23921c79
>2012-12-17 08:39:18,077 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:39:18,078 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25052668
>2012-12-17 08:39:18,097 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25052668
>2012-12-17 08:39:18,097 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 23921c79
>2012-12-17 08:39:18,097 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-17) [a2fcda8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:39:23,134 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3e6ad503] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:39:26,958 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [170be1f2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:39:27,065 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [2324f948] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:39:27,151 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2752da87
>2012-12-17 08:39:27,152 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3297ea5e
>2012-12-17 08:39:27,172 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [2324f948] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:39:27,321 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SpmStopVDSCommand, log id: 3297ea5e
>2012-12-17 08:39:27,346 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, ResetIrsVDSCommand, log id: 2752da87
>2012-12-17 08:39:27,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5a595460
>2012-12-17 08:39:27,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:39:27,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5a595460
>2012-12-17 08:39:33,340 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2f94bcb4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:39:37,424 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [1c47e0ef] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:39:37,453 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [1c47e0ef] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4609
>2012-12-17 08:39:37,458 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [1c47e0ef] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4609, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 34d2f5a1
>2012-12-17 08:39:37,481 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [1c47e0ef] spmStart polling started: taskId = f436c442-caa9-4d3e-a821-797b0b7349a5
>2012-12-17 08:39:38,504 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [1c47e0ef] spmStart polling ended: taskId = f436c442-caa9-4d3e-a821-797b0b7349a5 task status = finished
>2012-12-17 08:39:38,523 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [1c47e0ef] spmStart polling ended, spm status: SPM
>2012-12-17 08:39:38,525 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [1c47e0ef] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f436c442-caa9-4d3e-a821-797b0b7349a5), log id: 2d83166f
>2012-12-17 08:39:38,543 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [1c47e0ef] FINISH, HSMClearTaskVDSCommand, log id: 2d83166f
>2012-12-17 08:39:38,543 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [1c47e0ef] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7935e8c0, log id: 34d2f5a1
>2012-12-17 08:39:38,555 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [1c47e0ef] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:39:38,593 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [67020b95] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:39:38,593 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 30515c47
>2012-12-17 08:39:38,658 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [1c47e0ef] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:39:38,763 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [4595e4ad] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:39:38,803 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: dad526a
>2012-12-17 08:39:38,804 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 51114c57
>2012-12-17 08:39:38,824 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:39:38,966 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] FINISH, SpmStopVDSCommand, log id: 51114c57
>2012-12-17 08:39:38,980 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] FINISH, ResetIrsVDSCommand, log id: dad526a
>2012-12-17 08:39:38,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b1f2562
>2012-12-17 08:39:38,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [4595e4ad] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:39:38,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b1f2562
>2012-12-17 08:39:39,035 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:39:39,048 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4610
>2012-12-17 08:39:39,049 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4610, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5340a3f
>2012-12-17 08:39:39,071 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] spmStart polling started: taskId = 667c4ddd-9152-407e-8d5c-7c15b1c23ef2
>2012-12-17 08:39:40,090 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] spmStart polling ended: taskId = 667c4ddd-9152-407e-8d5c-7c15b1c23ef2 task status = finished
>2012-12-17 08:39:40,109 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] spmStart polling ended, spm status: SPM
>2012-12-17 08:39:40,111 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=667c4ddd-9152-407e-8d5c-7c15b1c23ef2), log id: 5b2a2182
>2012-12-17 08:39:40,129 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, HSMClearTaskVDSCommand, log id: 5b2a2182
>2012-12-17 08:39:40,129 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2afe0c90, log id: 5340a3f
>2012-12-17 08:39:40,165 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:39:40,207 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:39:40,208 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 14cbfe7d
>2012-12-17 08:39:40,227 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 14cbfe7d
>2012-12-17 08:39:40,227 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 30515c47
>2012-12-17 08:39:40,227 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [67020b95] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:39:40,227 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5cb0cb35
>2012-12-17 08:39:40,227 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:39:40,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1ab38693
>2012-12-17 08:39:40,247 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ab38693
>2012-12-17 08:39:40,247 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5cb0cb35
>2012-12-17 08:39:40,247 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [67020b95] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:39:43,558 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:39:49,082 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2edbd860] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:39:49,172 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-62) [2649fcd5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:39:49,213 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 37efd517
>2012-12-17 08:39:49,215 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7d9d856e
>2012-12-17 08:39:49,236 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:39:49,377 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, SpmStopVDSCommand, log id: 7d9d856e
>2012-12-17 08:39:49,393 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, ResetIrsVDSCommand, log id: 37efd517
>2012-12-17 08:39:49,403 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 649069a7
>2012-12-17 08:39:49,403 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2649fcd5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:39:49,403 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 649069a7
>2012-12-17 08:39:53,817 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:39:59,516 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [48aa9372] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:39:59,544 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [48aa9372] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4611
>2012-12-17 08:39:59,545 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4611, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 37150937
>2012-12-17 08:39:59,569 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] spmStart polling started: taskId = 76ae3316-adf5-4a61-9e31-e812274f0b81
>2012-12-17 08:40:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [436410e6] Checking autorecoverable hosts
>2012-12-17 08:40:00,003 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [436410e6] Autorecovering 0 hosts
>2012-12-17 08:40:00,003 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [436410e6] Checking autorecoverable hosts done
>2012-12-17 08:40:00,003 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [436410e6] Autorecovering storage domains is disabled, skipping
>2012-12-17 08:40:00,590 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] spmStart polling ended: taskId = 76ae3316-adf5-4a61-9e31-e812274f0b81 task status = finished
>2012-12-17 08:40:00,608 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] spmStart polling ended, spm status: SPM
>2012-12-17 08:40:00,609 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=76ae3316-adf5-4a61-9e31-e812274f0b81), log id: 2275fd22
>2012-12-17 08:40:00,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] FINISH, HSMClearTaskVDSCommand, log id: 2275fd22
>2012-12-17 08:40:00,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@703dd4da, log id: 37150937
>2012-12-17 08:40:01,089 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [48aa9372] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:40:01,493 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [7d3891a6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:40:01,493 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 796a43dc
>2012-12-17 08:40:01,580 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [48aa9372] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:40:01,725 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-22) [6e84ec96] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:40:01,790 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 71dad761
>2012-12-17 08:40:01,791 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: f94d5c2
>2012-12-17 08:40:01,811 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:01,963 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, SpmStopVDSCommand, log id: f94d5c2
>2012-12-17 08:40:01,999 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, ResetIrsVDSCommand, log id: 71dad761
>2012-12-17 08:40:02,011 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1066eaa
>2012-12-17 08:40:02,011 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:02,011 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1066eaa
>2012-12-17 08:40:02,056 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7d3891a6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:40:02,084 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7d3891a6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4612
>2012-12-17 08:40:02,086 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4612, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 56f2fb20
>2012-12-17 08:40:02,109 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] spmStart polling started: taskId = d371619b-1bc2-48b8-8617-e2b3cfa220f1
>2012-12-17 08:40:03,130 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] spmStart polling ended: taskId = d371619b-1bc2-48b8-8617-e2b3cfa220f1 task status = finished
>2012-12-17 08:40:03,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] spmStart polling ended, spm status: SPM
>2012-12-17 08:40:03,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d371619b-1bc2-48b8-8617-e2b3cfa220f1), log id: 34dd01c5
>2012-12-17 08:40:03,168 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, HSMClearTaskVDSCommand, log id: 34dd01c5
>2012-12-17 08:40:03,168 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@199f47a0, log id: 56f2fb20
>2012-12-17 08:40:03,187 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7d3891a6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:40:03,242 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:03,244 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4c3f6387
>2012-12-17 08:40:03,263 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4c3f6387
>2012-12-17 08:40:03,263 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 796a43dc
>2012-12-17 08:40:03,263 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [7d3891a6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:40:03,263 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 559f5b2c
>2012-12-17 08:40:03,263 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:03,265 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1dea2423
>2012-12-17 08:40:03,283 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1dea2423
>2012-12-17 08:40:03,284 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 559f5b2c
>2012-12-17 08:40:03,284 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [7d3891a6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:40:04,004 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [a2fcda8] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:40:12,093 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [50c114a6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:40:12,224 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [5231d6f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:40:12,279 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1883ac35
>2012-12-17 08:40:12,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 221fc926
>2012-12-17 08:40:12,300 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:12,460 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, SpmStopVDSCommand, log id: 221fc926
>2012-12-17 08:40:12,470 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, ResetIrsVDSCommand, log id: 1883ac35
>2012-12-17 08:40:12,481 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 12d0f2ba
>2012-12-17 08:40:12,481 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [5231d6f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:12,481 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 12d0f2ba
>2012-12-17 08:40:14,289 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:40:22,528 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4f93d023] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:40:22,554 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4f93d023] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4613
>2012-12-17 08:40:22,555 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4613, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 915e3c7
>2012-12-17 08:40:22,577 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] spmStart polling started: taskId = 159bc454-800f-4936-9e82-6bbd48ae5e7f
>2012-12-17 08:40:23,599 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] spmStart polling ended: taskId = 159bc454-800f-4936-9e82-6bbd48ae5e7f task status = finished
>2012-12-17 08:40:23,617 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] spmStart polling ended, spm status: SPM
>2012-12-17 08:40:23,618 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=159bc454-800f-4936-9e82-6bbd48ae5e7f), log id: 4543144a
>2012-12-17 08:40:23,637 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] FINISH, HSMClearTaskVDSCommand, log id: 4543144a
>2012-12-17 08:40:23,637 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@36e929cc, log id: 915e3c7
>2012-12-17 08:40:23,654 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4f93d023] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:40:23,709 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [3e5a5a03] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:40:23,709 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 46900d96
>2012-12-17 08:40:23,727 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4f93d023] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:40:23,863 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [37434c11] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:40:23,929 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [37434c11] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7feaddaa
>2012-12-17 08:40:23,931 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [37434c11] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 563e2af3
>2012-12-17 08:40:23,951 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [37434c11] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:24,345 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [37434c11] FINISH, SpmStopVDSCommand, log id: 563e2af3
>2012-12-17 08:40:24,379 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [37434c11] FINISH, ResetIrsVDSCommand, log id: 7feaddaa
>2012-12-17 08:40:24,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [37434c11] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4846c745
>2012-12-17 08:40:24,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [37434c11] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:24,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [37434c11] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4846c745
>2012-12-17 08:40:24,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:40:24,487 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4614
>2012-12-17 08:40:24,501 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4614, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 31bc37b2
>2012-12-17 08:40:24,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] spmStart polling started: taskId = b94467bc-d7a9-41a3-973b-fdd25d913fbe
>2012-12-17 08:40:25,547 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] spmStart polling ended: taskId = b94467bc-d7a9-41a3-973b-fdd25d913fbe task status = finished
>2012-12-17 08:40:25,565 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] spmStart polling ended, spm status: SPM
>2012-12-17 08:40:25,566 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b94467bc-d7a9-41a3-973b-fdd25d913fbe), log id: 5906b6c7
>2012-12-17 08:40:25,584 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, HSMClearTaskVDSCommand, log id: 5906b6c7
>2012-12-17 08:40:25,584 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@47c9f07a, log id: 31bc37b2
>2012-12-17 08:40:25,904 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:40:25,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:25,985 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 46d6d585
>2012-12-17 08:40:26,004 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 46d6d585
>2012-12-17 08:40:26,004 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 46900d96
>2012-12-17 08:40:26,004 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [3e5a5a03] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:40:26,005 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 425c3318
>2012-12-17 08:40:26,005 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:26,006 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3ca213ac
>2012-12-17 08:40:26,030 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3ca213ac
>2012-12-17 08:40:26,030 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 425c3318
>2012-12-17 08:40:26,030 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [3e5a5a03] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:40:34,513 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [5a98cdcf] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:40:34,632 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [71e1da67] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:40:34,749 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2decfdeb
>2012-12-17 08:40:34,751 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5a492037
>2012-12-17 08:40:34,755 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:40:34,773 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:35,350 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] FINISH, SpmStopVDSCommand, log id: 5a492037
>2012-12-17 08:40:35,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] FINISH, ResetIrsVDSCommand, log id: 2decfdeb
>2012-12-17 08:40:35,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d2bb9a5
>2012-12-17 08:40:35,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [71e1da67] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:35,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d2bb9a5
>2012-12-17 08:40:46,074 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [57e0db40] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:40:46,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [57e0db40] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4615
>2012-12-17 08:40:46,417 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4615, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 243f8158
>2012-12-17 08:40:46,441 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] spmStart polling started: taskId = 7eeba601-5d2c-42cf-ae10-56ca5a93975d
>2012-12-17 08:40:47,462 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] spmStart polling ended: taskId = 7eeba601-5d2c-42cf-ae10-56ca5a93975d task status = finished
>2012-12-17 08:40:47,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] spmStart polling ended, spm status: SPM
>2012-12-17 08:40:47,482 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7eeba601-5d2c-42cf-ae10-56ca5a93975d), log id: 3a031217
>2012-12-17 08:40:47,501 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] FINISH, HSMClearTaskVDSCommand, log id: 3a031217
>2012-12-17 08:40:47,501 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1f98c1c2, log id: 243f8158
>2012-12-17 08:40:47,512 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [57e0db40] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:40:47,562 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:40:47,562 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6bcbc7af
>2012-12-17 08:40:47,573 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [57e0db40] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:40:47,724 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [68b5cbc1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:40:47,806 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 131827
>2012-12-17 08:40:47,807 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 266a42c6
>2012-12-17 08:40:47,828 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:48,005 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] FINISH, SpmStopVDSCommand, log id: 266a42c6
>2012-12-17 08:40:48,040 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] FINISH, ResetIrsVDSCommand, log id: 131827
>2012-12-17 08:40:48,051 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 43580ec8
>2012-12-17 08:40:48,051 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [68b5cbc1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:48,051 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 43580ec8
>2012-12-17 08:40:48,128 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:40:48,138 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4616
>2012-12-17 08:40:48,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4616, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 300e7058
>2012-12-17 08:40:48,182 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling started: taskId = 99d4ef12-f3cb-4b9c-8a24-5645096ca296
>2012-12-17 08:40:49,204 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended: taskId = 99d4ef12-f3cb-4b9c-8a24-5645096ca296 task status = finished
>2012-12-17 08:40:49,222 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended, spm status: SPM
>2012-12-17 08:40:49,223 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=99d4ef12-f3cb-4b9c-8a24-5645096ca296), log id: 4cb5bbcf
>2012-12-17 08:40:49,242 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMClearTaskVDSCommand, log id: 4cb5bbcf
>2012-12-17 08:40:49,243 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2f3d3a61, log id: 300e7058
>2012-12-17 08:40:49,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:40:49,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:49,330 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7a59be09
>2012-12-17 08:40:49,350 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7a59be09
>2012-12-17 08:40:49,350 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6bcbc7af
>2012-12-17 08:40:49,350 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:40:49,350 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7f27ee9a
>2012-12-17 08:40:49,350 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:49,352 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 63d517b7
>2012-12-17 08:40:49,371 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 63d517b7
>2012-12-17 08:40:49,371 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7f27ee9a
>2012-12-17 08:40:49,371 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:40:55,115 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [95f5063] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:40:58,181 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3e6ad503] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:40:58,286 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [7f520532] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:40:58,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7304e581
>2012-12-17 08:40:58,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1d7861cf
>2012-12-17 08:40:58,497 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [7f520532] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:58,672 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SpmStopVDSCommand, log id: 1d7861cf
>2012-12-17 08:40:58,687 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, ResetIrsVDSCommand, log id: 7304e581
>2012-12-17 08:40:58,698 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 94053ea
>2012-12-17 08:40:58,698 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:40:58,698 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 94053ea
>2012-12-17 08:41:06,529 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:41:08,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [41c296ff] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:41:08,778 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [41c296ff] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4617
>2012-12-17 08:41:08,779 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [41c296ff] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4617, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 48b82caa
>2012-12-17 08:41:08,803 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [41c296ff] spmStart polling started: taskId = 8b6bd3d9-d57e-415d-a220-437449316252
>2012-12-17 08:41:09,825 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [41c296ff] spmStart polling ended: taskId = 8b6bd3d9-d57e-415d-a220-437449316252 task status = finished
>2012-12-17 08:41:09,843 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [41c296ff] spmStart polling ended, spm status: SPM
>2012-12-17 08:41:09,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [41c296ff] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8b6bd3d9-d57e-415d-a220-437449316252), log id: 5508d3f2
>2012-12-17 08:41:09,863 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [41c296ff] FINISH, HSMClearTaskVDSCommand, log id: 5508d3f2
>2012-12-17 08:41:09,863 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [41c296ff] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3692e07b, log id: 48b82caa
>2012-12-17 08:41:09,877 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [41c296ff] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:41:09,921 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [181d86f1] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:41:09,921 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2d23e813
>2012-12-17 08:41:09,946 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [41c296ff] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:41:10,034 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [6c95e244] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:41:10,089 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [6c95e244] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 16f2d2bb
>2012-12-17 08:41:10,091 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [6c95e244] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1e9038cc
>2012-12-17 08:41:10,150 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [6c95e244] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:10,302 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [6c95e244] FINISH, SpmStopVDSCommand, log id: 1e9038cc
>2012-12-17 08:41:10,325 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [6c95e244] FINISH, ResetIrsVDSCommand, log id: 16f2d2bb
>2012-12-17 08:41:10,336 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [6c95e244] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 9efdf6b
>2012-12-17 08:41:10,336 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [6c95e244] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:10,336 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [6c95e244] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 9efdf6b
>2012-12-17 08:41:11,087 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:41:11,102 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4618
>2012-12-17 08:41:11,103 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4618, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 63a20d01
>2012-12-17 08:41:11,125 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling started: taskId = 112a9648-b185-44ca-a2b8-349fc0d0bafa
>2012-12-17 08:41:12,151 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling ended: taskId = 112a9648-b185-44ca-a2b8-349fc0d0bafa task status = finished
>2012-12-17 08:41:12,169 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling ended, spm status: SPM
>2012-12-17 08:41:12,171 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=112a9648-b185-44ca-a2b8-349fc0d0bafa), log id: 5e05cf34
>2012-12-17 08:41:12,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, HSMClearTaskVDSCommand, log id: 5e05cf34
>2012-12-17 08:41:12,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f668f25, log id: 63a20d01
>2012-12-17 08:41:12,209 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:41:12,236 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:12,237 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 20465ca
>2012-12-17 08:41:12,258 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 20465ca
>2012-12-17 08:41:12,258 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2d23e813
>2012-12-17 08:41:12,258 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [181d86f1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:41:12,259 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6eb6b1ce
>2012-12-17 08:41:12,259 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:12,260 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d3639a1
>2012-12-17 08:41:12,279 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d3639a1
>2012-12-17 08:41:12,279 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6eb6b1ce
>2012-12-17 08:41:12,279 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [181d86f1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:41:17,089 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [25db9572] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:41:21,325 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [48fde2a8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:41:21,606 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [28e17066] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:41:21,657 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [28e17066] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 20dbea2c
>2012-12-17 08:41:21,659 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [28e17066] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 45724812
>2012-12-17 08:41:21,679 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [28e17066] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:21,853 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [28e17066] FINISH, SpmStopVDSCommand, log id: 45724812
>2012-12-17 08:41:21,916 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [28e17066] FINISH, ResetIrsVDSCommand, log id: 20dbea2c
>2012-12-17 08:41:21,927 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [28e17066] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: bbad73b
>2012-12-17 08:41:21,927 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [28e17066] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:21,927 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [28e17066] FINISH, MarkPoolInReconstructModeVDSCommand, log id: bbad73b
>2012-12-17 08:41:27,275 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:41:32,030 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [50db5e16] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:41:32,040 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [50db5e16] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4619
>2012-12-17 08:41:32,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4619, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 38f4581d
>2012-12-17 08:41:32,064 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] spmStart polling started: taskId = 9e744fc2-fb9e-4b53-be59-d8db23236c4f
>2012-12-17 08:41:33,085 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] spmStart polling ended: taskId = 9e744fc2-fb9e-4b53-be59-d8db23236c4f task status = finished
>2012-12-17 08:41:33,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] spmStart polling ended, spm status: SPM
>2012-12-17 08:41:33,105 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9e744fc2-fb9e-4b53-be59-d8db23236c4f), log id: a079d55
>2012-12-17 08:41:33,125 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] FINISH, HSMClearTaskVDSCommand, log id: a079d55
>2012-12-17 08:41:33,125 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@79ea3c97, log id: 38f4581d
>2012-12-17 08:41:33,137 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [50db5e16] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:41:33,176 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:41:33,176 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3345386a
>2012-12-17 08:41:33,194 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [50db5e16] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:41:33,423 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [51589818] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:41:33,500 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [51589818] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 21df1f84
>2012-12-17 08:41:33,501 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [51589818] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2ba9a65f
>2012-12-17 08:41:33,522 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [51589818] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:33,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [51589818] FINISH, SpmStopVDSCommand, log id: 2ba9a65f
>2012-12-17 08:41:33,710 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [51589818] FINISH, ResetIrsVDSCommand, log id: 21df1f84
>2012-12-17 08:41:33,721 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [51589818] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 39c103a5
>2012-12-17 08:41:33,721 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [51589818] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:33,721 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [51589818] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 39c103a5
>2012-12-17 08:41:33,790 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:41:33,838 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4620
>2012-12-17 08:41:33,839 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4620, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 368f4227
>2012-12-17 08:41:33,861 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling started: taskId = abfe3d5b-6471-49bf-93fa-5e545f014038
>2012-12-17 08:41:34,883 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling ended: taskId = abfe3d5b-6471-49bf-93fa-5e545f014038 task status = finished
>2012-12-17 08:41:34,902 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling ended, spm status: SPM
>2012-12-17 08:41:34,903 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=abfe3d5b-6471-49bf-93fa-5e545f014038), log id: 44803a1a
>2012-12-17 08:41:34,921 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMClearTaskVDSCommand, log id: 44803a1a
>2012-12-17 08:41:34,921 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e66a418, log id: 368f4227
>2012-12-17 08:41:34,930 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:41:34,966 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:34,970 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3d327944
>2012-12-17 08:41:34,989 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3d327944
>2012-12-17 08:41:34,989 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3345386a
>2012-12-17 08:41:34,989 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:41:34,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 32e24cbd
>2012-12-17 08:41:34,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:34,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 79b72005
>2012-12-17 08:41:35,009 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 79b72005
>2012-12-17 08:41:35,009 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 32e24cbd
>2012-12-17 08:41:35,009 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:41:37,468 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:41:43,831 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [95f5063] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:41:43,961 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-100) [6bb29080] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:41:44,020 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 70470419
>2012-12-17 08:41:44,021 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1e2409aa
>2012-12-17 08:41:44,041 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:44,195 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SpmStopVDSCommand, log id: 1e2409aa
>2012-12-17 08:41:44,209 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, ResetIrsVDSCommand, log id: 70470419
>2012-12-17 08:41:44,220 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3fa0fff0
>2012-12-17 08:41:44,220 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:44,220 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3fa0fff0
>2012-12-17 08:41:47,703 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:41:54,320 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:41:54,333 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4621
>2012-12-17 08:41:54,334 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4621, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2d457424
>2012-12-17 08:41:54,357 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] spmStart polling started: taskId = a44e1167-47d2-4270-868f-f49c5f62a28b
>2012-12-17 08:41:55,379 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] spmStart polling ended: taskId = a44e1167-47d2-4270-868f-f49c5f62a28b task status = finished
>2012-12-17 08:41:55,397 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] spmStart polling ended, spm status: SPM
>2012-12-17 08:41:55,398 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a44e1167-47d2-4270-868f-f49c5f62a28b), log id: 7193a3ac
>2012-12-17 08:41:55,417 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, HSMClearTaskVDSCommand, log id: 7193a3ac
>2012-12-17 08:41:55,417 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@13deab75, log id: 2d457424
>2012-12-17 08:41:55,819 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:41:56,023 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [2f94bcb4] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:41:56,023 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 42a64369
>2012-12-17 08:41:56,055 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:41:56,197 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-85) [1d8c274e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:41:56,279 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 65fff724
>2012-12-17 08:41:56,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 799b3086
>2012-12-17 08:41:56,302 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:56,434 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, SpmStopVDSCommand, log id: 799b3086
>2012-12-17 08:41:56,452 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, ResetIrsVDSCommand, log id: 65fff724
>2012-12-17 08:41:56,463 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6319a1a5
>2012-12-17 08:41:56,463 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:56,463 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6319a1a5
>2012-12-17 08:41:56,511 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2f94bcb4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:41:56,520 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2f94bcb4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4622
>2012-12-17 08:41:56,522 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4622, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 582723d1
>2012-12-17 08:41:56,543 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] spmStart polling started: taskId = 485a87ec-e214-41fe-9006-020eb5d3cab0
>2012-12-17 08:41:57,565 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] spmStart polling ended: taskId = 485a87ec-e214-41fe-9006-020eb5d3cab0 task status = finished
>2012-12-17 08:41:57,583 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] spmStart polling ended, spm status: SPM
>2012-12-17 08:41:57,584 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=485a87ec-e214-41fe-9006-020eb5d3cab0), log id: 408c1872
>2012-12-17 08:41:57,602 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, HSMClearTaskVDSCommand, log id: 408c1872
>2012-12-17 08:41:57,602 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2d990579, log id: 582723d1
>2012-12-17 08:41:57,649 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2f94bcb4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:41:57,667 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:57,669 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 63435e95
>2012-12-17 08:41:57,690 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 63435e95
>2012-12-17 08:41:57,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 42a64369
>2012-12-17 08:41:57,690 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [2f94bcb4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:41:57,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5bf6bb1c
>2012-12-17 08:41:57,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:41:57,691 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7ae2eb8e
>2012-12-17 08:41:57,710 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7ae2eb8e
>2012-12-17 08:41:57,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5bf6bb1c
>2012-12-17 08:41:57,711 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [2f94bcb4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:41:57,942 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [6c95e244] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:42:06,610 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:42:06,861 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-68) [6acb626c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:42:06,927 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [6acb626c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5a4f4afc
>2012-12-17 08:42:06,929 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [6acb626c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 797b5b5a
>2012-12-17 08:42:06,949 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [6acb626c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:42:07,117 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [6acb626c] FINISH, SpmStopVDSCommand, log id: 797b5b5a
>2012-12-17 08:42:07,142 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [6acb626c] FINISH, ResetIrsVDSCommand, log id: 5a4f4afc
>2012-12-17 08:42:07,153 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [6acb626c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 118304c7
>2012-12-17 08:42:07,153 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [6acb626c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:42:07,153 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [6acb626c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 118304c7
>2012-12-17 08:42:08,129 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:42:17,232 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [2cd85c0e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:42:17,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [2cd85c0e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4623
>2012-12-17 08:42:17,288 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [2cd85c0e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4623, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4237ed0f
>2012-12-17 08:42:17,312 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [2cd85c0e] spmStart polling started: taskId = e8b915c4-d475-4737-80b6-0f1f78ad2a55
>2012-12-17 08:42:18,334 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [2cd85c0e] spmStart polling ended: taskId = e8b915c4-d475-4737-80b6-0f1f78ad2a55 task status = finished
>2012-12-17 08:42:18,351 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [2cd85c0e] spmStart polling ended, spm status: SPM
>2012-12-17 08:42:18,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [2cd85c0e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e8b915c4-d475-4737-80b6-0f1f78ad2a55), log id: 19ea12c4
>2012-12-17 08:42:18,371 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [2cd85c0e] FINISH, HSMClearTaskVDSCommand, log id: 19ea12c4
>2012-12-17 08:42:18,371 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [2cd85c0e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@56760f26, log id: 4237ed0f
>2012-12-17 08:42:18,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [2cd85c0e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:42:18,445 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [1f4324c3] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:42:18,445 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5016d94d
>2012-12-17 08:42:18,467 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [2cd85c0e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:42:18,702 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-4) [5f1742db] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:42:18,777 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2e95e1e3
>2012-12-17 08:42:18,779 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 30f90f68
>2012-12-17 08:42:18,798 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:42:18,921 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] FINISH, SpmStopVDSCommand, log id: 30f90f68
>2012-12-17 08:42:18,940 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] FINISH, ResetIrsVDSCommand, log id: 2e95e1e3
>2012-12-17 08:42:18,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c63e19c
>2012-12-17 08:42:18,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5f1742db] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:42:18,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c63e19c
>2012-12-17 08:42:18,999 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:42:19,007 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4624
>2012-12-17 08:42:19,008 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4624, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 10a1d5c4
>2012-12-17 08:42:19,030 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] spmStart polling started: taskId = 41f26e2a-9d26-4f4e-8a83-c0f643520a84
>2012-12-17 08:42:20,052 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] spmStart polling ended: taskId = 41f26e2a-9d26-4f4e-8a83-c0f643520a84 task status = finished
>2012-12-17 08:42:20,070 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] spmStart polling ended, spm status: SPM
>2012-12-17 08:42:20,072 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=41f26e2a-9d26-4f4e-8a83-c0f643520a84), log id: 7bf9e7d6
>2012-12-17 08:42:20,090 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, HSMClearTaskVDSCommand, log id: 7bf9e7d6
>2012-12-17 08:42:20,090 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@53e3d230, log id: 10a1d5c4
>2012-12-17 08:42:20,105 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:42:20,123 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:42:20,124 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 11f4d41b
>2012-12-17 08:42:20,146 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 11f4d41b
>2012-12-17 08:42:20,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5016d94d
>2012-12-17 08:42:20,146 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [1f4324c3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:42:20,147 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 60787385
>2012-12-17 08:42:20,147 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:42:20,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 46b705b6
>2012-12-17 08:42:20,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 46b705b6
>2012-12-17 08:42:20,167 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 60787385
>2012-12-17 08:42:20,167 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [1f4324c3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:42:28,521 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:42:29,034 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [a2fcda8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:42:29,161 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [27478f27] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:42:29,204 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [27478f27] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 19ad8938
>2012-12-17 08:42:29,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [27478f27] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 335c39ca
>2012-12-17 08:42:29,225 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [27478f27] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:42:29,367 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [27478f27] FINISH, SpmStopVDSCommand, log id: 335c39ca
>2012-12-17 08:42:29,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [27478f27] FINISH, ResetIrsVDSCommand, log id: 19ad8938
>2012-12-17 08:42:29,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [27478f27] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5d4ce839
>2012-12-17 08:42:29,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [27478f27] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:42:29,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [27478f27] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5d4ce839
>2012-12-17 08:42:38,710 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [5231d6f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:42:39,443 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7543fd1a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:42:39,457 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7543fd1a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4625
>2012-12-17 08:42:39,458 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4625, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6a7974bc
>2012-12-17 08:42:39,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] spmStart polling started: taskId = 926350d7-83d9-4e68-9868-3f2732042968
>2012-12-17 08:42:40,503 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] spmStart polling ended: taskId = 926350d7-83d9-4e68-9868-3f2732042968 task status = finished
>2012-12-17 08:42:40,520 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] spmStart polling ended, spm status: SPM
>2012-12-17 08:42:40,522 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=926350d7-83d9-4e68-9868-3f2732042968), log id: 58587d6b
>2012-12-17 08:42:40,540 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, HSMClearTaskVDSCommand, log id: 58587d6b
>2012-12-17 08:42:40,540 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1992291d, log id: 6a7974bc
>2012-12-17 08:42:40,968 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7543fd1a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:42:41,078 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [41698d74] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:42:41,078 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 478d8b67
>2012-12-17 08:42:41,106 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7543fd1a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:42:41,214 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [e4c8c37] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:42:41,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5f021e29
>2012-12-17 08:42:41,266 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4d1a071e
>2012-12-17 08:42:41,286 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:42:41,681 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, SpmStopVDSCommand, log id: 4d1a071e
>2012-12-17 08:42:41,698 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, ResetIrsVDSCommand, log id: 5f021e29
>2012-12-17 08:42:41,714 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24202844
>2012-12-17 08:42:41,714 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:42:41,714 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24202844
>2012-12-17 08:42:41,789 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [41698d74] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:42:41,801 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [41698d74] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4626
>2012-12-17 08:42:41,802 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4626, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 77c97917
>2012-12-17 08:42:41,824 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [41698d74] spmStart polling started: taskId = 1e173aa2-fd41-4a03-873f-8401c8333a5f
>2012-12-17 08:42:42,844 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [41698d74] spmStart polling ended: taskId = 1e173aa2-fd41-4a03-873f-8401c8333a5f task status = finished
>2012-12-17 08:42:42,862 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [41698d74] spmStart polling ended, spm status: SPM
>2012-12-17 08:42:42,863 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1e173aa2-fd41-4a03-873f-8401c8333a5f), log id: 466f3d5c
>2012-12-17 08:42:42,882 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, HSMClearTaskVDSCommand, log id: 466f3d5c
>2012-12-17 08:42:42,882 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31258724, log id: 77c97917
>2012-12-17 08:42:42,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [41698d74] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:42:42,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:42:42,911 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d5dd288
>2012-12-17 08:42:42,931 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d5dd288
>2012-12-17 08:42:42,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 478d8b67
>2012-12-17 08:42:42,931 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [41698d74] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:42:42,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 30fe0033
>2012-12-17 08:42:42,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:42:42,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3287c588
>2012-12-17 08:42:42,951 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3287c588
>2012-12-17 08:42:42,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 30fe0033
>2012-12-17 08:42:42,951 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [41698d74] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:42:48,916 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [778f4477] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:42:51,828 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [25db9572] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:42:51,936 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [34599a45] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:42:52,000 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [34599a45] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 19ffdefa
>2012-12-17 08:42:52,002 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [34599a45] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4b547485
>2012-12-17 08:42:52,022 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [34599a45] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:42:52,175 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [34599a45] FINISH, SpmStopVDSCommand, log id: 4b547485
>2012-12-17 08:42:52,188 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [34599a45] FINISH, ResetIrsVDSCommand, log id: 19ffdefa
>2012-12-17 08:42:52,199 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [34599a45] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: d5070bb
>2012-12-17 08:42:52,199 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [34599a45] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:42:52,199 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [34599a45] FINISH, MarkPoolInReconstructModeVDSCommand, log id: d5070bb
>2012-12-17 08:42:59,171 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:43:02,247 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [5fbba580] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:43:02,339 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [5fbba580] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4627
>2012-12-17 08:43:02,341 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4627, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5f548a68
>2012-12-17 08:43:02,365 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] spmStart polling started: taskId = f9dba6c0-56c0-4914-aa0a-1a8b584713a9
>2012-12-17 08:43:03,390 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] spmStart polling ended: taskId = f9dba6c0-56c0-4914-aa0a-1a8b584713a9 task status = finished
>2012-12-17 08:43:03,408 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] spmStart polling ended, spm status: SPM
>2012-12-17 08:43:03,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f9dba6c0-56c0-4914-aa0a-1a8b584713a9), log id: 3075ad3c
>2012-12-17 08:43:03,428 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] FINISH, HSMClearTaskVDSCommand, log id: 3075ad3c
>2012-12-17 08:43:03,428 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7dc056fe, log id: 5f548a68
>2012-12-17 08:43:03,477 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [5fbba580] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:43:03,580 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [419474a0] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:43:03,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 29025348
>2012-12-17 08:43:03,626 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [5fbba580] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:43:03,750 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-92) [30867547] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:43:03,797 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [30867547] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5b39a81a
>2012-12-17 08:43:03,798 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [30867547] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3ac6f7b6
>2012-12-17 08:43:03,818 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [30867547] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:03,940 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [30867547] FINISH, SpmStopVDSCommand, log id: 3ac6f7b6
>2012-12-17 08:43:03,956 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [30867547] FINISH, ResetIrsVDSCommand, log id: 5b39a81a
>2012-12-17 08:43:03,967 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [30867547] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ef2f7d
>2012-12-17 08:43:03,967 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [30867547] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:03,967 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [30867547] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ef2f7d
>2012-12-17 08:43:04,063 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:43:04,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4628
>2012-12-17 08:43:04,073 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4628, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6ccb66f7
>2012-12-17 08:43:04,095 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] spmStart polling started: taskId = 0bcdc6fe-1fa5-40cc-a25c-055de6025850
>2012-12-17 08:43:05,117 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] spmStart polling ended: taskId = 0bcdc6fe-1fa5-40cc-a25c-055de6025850 task status = finished
>2012-12-17 08:43:05,135 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] spmStart polling ended, spm status: SPM
>2012-12-17 08:43:05,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0bcdc6fe-1fa5-40cc-a25c-055de6025850), log id: 6ad96cf9
>2012-12-17 08:43:05,155 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, HSMClearTaskVDSCommand, log id: 6ad96cf9
>2012-12-17 08:43:05,155 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6881863c, log id: 6ccb66f7
>2012-12-17 08:43:05,173 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:43:05,195 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:05,196 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1d24c978
>2012-12-17 08:43:05,217 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1d24c978
>2012-12-17 08:43:05,217 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 29025348
>2012-12-17 08:43:05,217 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [419474a0] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:43:05,217 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5a26f43a
>2012-12-17 08:43:05,217 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:05,219 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 51b5fa83
>2012-12-17 08:43:05,252 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 51b5fa83
>2012-12-17 08:43:05,252 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5a26f43a
>2012-12-17 08:43:05,252 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [419474a0] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:43:09,387 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [51589818] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:43:14,103 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [42bc2eb6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:43:14,227 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [49c4851d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:43:14,419 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 20b0ed13
>2012-12-17 08:43:14,421 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5db5d8df
>2012-12-17 08:43:14,443 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:14,583 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] FINISH, SpmStopVDSCommand, log id: 5db5d8df
>2012-12-17 08:43:14,661 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] FINISH, ResetIrsVDSCommand, log id: 20b0ed13
>2012-12-17 08:43:14,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1b741b19
>2012-12-17 08:43:14,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [49c4851d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:14,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1b741b19
>2012-12-17 08:43:19,594 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:43:24,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [653dcf69] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:43:24,763 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [653dcf69] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4629
>2012-12-17 08:43:24,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4629, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1dff7354
>2012-12-17 08:43:24,788 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] spmStart polling started: taskId = b6cff0ee-6f1f-4124-ad17-1bb770f85473
>2012-12-17 08:43:25,813 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] spmStart polling ended: taskId = b6cff0ee-6f1f-4124-ad17-1bb770f85473 task status = finished
>2012-12-17 08:43:25,831 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] spmStart polling ended, spm status: SPM
>2012-12-17 08:43:25,832 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b6cff0ee-6f1f-4124-ad17-1bb770f85473), log id: 1b3876c1
>2012-12-17 08:43:25,850 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] FINISH, HSMClearTaskVDSCommand, log id: 1b3876c1
>2012-12-17 08:43:25,850 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@17fc86dd, log id: 1dff7354
>2012-12-17 08:43:26,008 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [653dcf69] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:43:26,077 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:43:26,078 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 21707aaa
>2012-12-17 08:43:26,098 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [653dcf69] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:43:26,212 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [3eec3b83] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:43:26,359 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 50bdc77c
>2012-12-17 08:43:26,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7eb94769
>2012-12-17 08:43:26,380 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:26,519 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] FINISH, SpmStopVDSCommand, log id: 7eb94769
>2012-12-17 08:43:26,534 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] FINISH, ResetIrsVDSCommand, log id: 50bdc77c
>2012-12-17 08:43:26,544 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5bf60e71
>2012-12-17 08:43:26,544 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3eec3b83] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:26,544 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5bf60e71
>2012-12-17 08:43:26,614 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:43:26,624 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4630
>2012-12-17 08:43:26,625 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4630, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: c74ee7f
>2012-12-17 08:43:26,651 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling started: taskId = b26927ca-3904-42e4-b775-f99a5105914f
>2012-12-17 08:43:27,673 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling ended: taskId = b26927ca-3904-42e4-b775-f99a5105914f task status = finished
>2012-12-17 08:43:27,702 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling ended, spm status: SPM
>2012-12-17 08:43:27,704 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b26927ca-3904-42e4-b775-f99a5105914f), log id: 4f39fc82
>2012-12-17 08:43:27,722 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMClearTaskVDSCommand, log id: 4f39fc82
>2012-12-17 08:43:27,722 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@67f0e370, log id: c74ee7f
>2012-12-17 08:43:27,734 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:43:27,748 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:27,750 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5ff3cf0c
>2012-12-17 08:43:27,769 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5ff3cf0c
>2012-12-17 08:43:27,769 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 21707aaa
>2012-12-17 08:43:27,769 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:43:27,769 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7f9e4e0a
>2012-12-17 08:43:27,770 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:27,771 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: a4ef26f
>2012-12-17 08:43:27,791 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: a4ef26f
>2012-12-17 08:43:27,791 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7f9e4e0a
>2012-12-17 08:43:27,791 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:43:29,858 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:43:36,658 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [778f4477] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:43:36,818 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [41dfde4b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:43:37,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1bd9ac11
>2012-12-17 08:43:37,048 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 91402ba
>2012-12-17 08:43:37,068 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:37,232 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] FINISH, SpmStopVDSCommand, log id: 91402ba
>2012-12-17 08:43:37,260 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] FINISH, ResetIrsVDSCommand, log id: 1bd9ac11
>2012-12-17 08:43:37,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1864106e
>2012-12-17 08:43:37,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [41dfde4b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:37,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1864106e
>2012-12-17 08:43:40,070 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:43:47,324 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:43:47,365 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4631
>2012-12-17 08:43:47,366 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4631, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5c14c7b1
>2012-12-17 08:43:47,390 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] spmStart polling started: taskId = a8d01c5a-dcb7-4094-8858-771fd6a48f88
>2012-12-17 08:43:48,411 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] spmStart polling ended: taskId = a8d01c5a-dcb7-4094-8858-771fd6a48f88 task status = finished
>2012-12-17 08:43:48,428 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] spmStart polling ended, spm status: SPM
>2012-12-17 08:43:48,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a8d01c5a-dcb7-4094-8858-771fd6a48f88), log id: 427b5e80
>2012-12-17 08:43:48,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, HSMClearTaskVDSCommand, log id: 427b5e80
>2012-12-17 08:43:48,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31794799, log id: 5c14c7b1
>2012-12-17 08:43:48,467 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:43:48,509 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [2649fcd5] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:43:48,509 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 79fbfef
>2012-12-17 08:43:48,534 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:43:48,688 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [5ad4c70] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:43:48,747 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [5ad4c70] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5f28bffc
>2012-12-17 08:43:48,749 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [5ad4c70] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3010976e
>2012-12-17 08:43:48,768 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [5ad4c70] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:49,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [5ad4c70] FINISH, SpmStopVDSCommand, log id: 3010976e
>2012-12-17 08:43:49,485 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [5ad4c70] FINISH, ResetIrsVDSCommand, log id: 5f28bffc
>2012-12-17 08:43:49,495 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [5ad4c70] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5c85be4e
>2012-12-17 08:43:49,495 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5ad4c70] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:49,496 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [5ad4c70] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5c85be4e
>2012-12-17 08:43:49,550 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2649fcd5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:43:49,564 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2649fcd5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4632
>2012-12-17 08:43:49,565 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4632, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 20c10828
>2012-12-17 08:43:49,589 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] spmStart polling started: taskId = 85785197-903d-43d8-9ec7-a9980e5fd689
>2012-12-17 08:43:51,143 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] spmStart polling ended: taskId = 85785197-903d-43d8-9ec7-a9980e5fd689 task status = finished
>2012-12-17 08:43:51,161 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] spmStart polling ended, spm status: SPM
>2012-12-17 08:43:51,163 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=85785197-903d-43d8-9ec7-a9980e5fd689), log id: 338d1d56
>2012-12-17 08:43:51,186 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, HSMClearTaskVDSCommand, log id: 338d1d56
>2012-12-17 08:43:51,186 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@71af44c5, log id: 20c10828
>2012-12-17 08:43:51,324 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2649fcd5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:43:51,345 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:51,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 16ce6d05
>2012-12-17 08:43:51,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 16ce6d05
>2012-12-17 08:43:51,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 79fbfef
>2012-12-17 08:43:51,368 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [2649fcd5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:43:51,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4a576ef5
>2012-12-17 08:43:51,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:43:51,369 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7ed06cd7
>2012-12-17 08:43:51,389 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7ed06cd7
>2012-12-17 08:43:51,389 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4a576ef5
>2012-12-17 08:43:51,389 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [2649fcd5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:43:59,598 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [16c2b2c4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:43:59,736 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [40a5fc59] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:43:59,786 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2e0adfd
>2012-12-17 08:43:59,788 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5440ad2a
>2012-12-17 08:43:59,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:00,190 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, SpmStopVDSCommand, log id: 5440ad2a
>2012-12-17 08:44:00,222 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, ResetIrsVDSCommand, log id: 2e0adfd
>2012-12-17 08:44:00,233 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: efbf806
>2012-12-17 08:44:00,233 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [40a5fc59] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:00,233 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, MarkPoolInReconstructModeVDSCommand, log id: efbf806
>2012-12-17 08:44:01,221 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:44:10,432 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1daf49b5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:44:10,511 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1daf49b5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4633
>2012-12-17 08:44:10,512 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4633, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 25316b5e
>2012-12-17 08:44:10,534 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] spmStart polling started: taskId = 16704ac1-74c8-4e6c-b523-11d760e37cf5
>2012-12-17 08:44:11,564 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] spmStart polling ended: taskId = 16704ac1-74c8-4e6c-b523-11d760e37cf5 task status = finished
>2012-12-17 08:44:11,582 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] spmStart polling ended, spm status: SPM
>2012-12-17 08:44:11,584 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=16704ac1-74c8-4e6c-b523-11d760e37cf5), log id: b786629
>2012-12-17 08:44:11,607 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] FINISH, HSMClearTaskVDSCommand, log id: b786629
>2012-12-17 08:44:11,607 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5c291a4d, log id: 25316b5e
>2012-12-17 08:44:11,640 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1daf49b5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:44:11,699 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:44:11,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 787e9bf5
>2012-12-17 08:44:11,729 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1daf49b5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:44:11,836 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [61acfe5d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:44:11,893 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5ee48e16
>2012-12-17 08:44:11,894 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 14d8d12d
>2012-12-17 08:44:11,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:12,117 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] FINISH, SpmStopVDSCommand, log id: 14d8d12d
>2012-12-17 08:44:12,131 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] FINISH, ResetIrsVDSCommand, log id: 5ee48e16
>2012-12-17 08:44:12,142 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ca04d2a
>2012-12-17 08:44:12,142 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61acfe5d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:12,142 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ca04d2a
>2012-12-17 08:44:12,209 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:44:12,217 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4634
>2012-12-17 08:44:12,218 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4634, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: bd20f88
>2012-12-17 08:44:12,240 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling started: taskId = c4a4c340-eeaf-4874-8e45-c62172294337
>2012-12-17 08:44:13,266 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling ended: taskId = c4a4c340-eeaf-4874-8e45-c62172294337 task status = finished
>2012-12-17 08:44:13,285 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling ended, spm status: SPM
>2012-12-17 08:44:13,287 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c4a4c340-eeaf-4874-8e45-c62172294337), log id: 6c334f1b
>2012-12-17 08:44:13,306 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMClearTaskVDSCommand, log id: 6c334f1b
>2012-12-17 08:44:13,307 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c8ead1c, log id: bd20f88
>2012-12-17 08:44:13,345 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:44:13,386 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:13,388 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 27090873
>2012-12-17 08:44:13,442 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 27090873
>2012-12-17 08:44:13,442 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 787e9bf5
>2012-12-17 08:44:13,442 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:44:13,443 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 740bd1e2
>2012-12-17 08:44:13,443 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:13,445 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 76ab3960
>2012-12-17 08:44:13,472 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 76ab3960
>2012-12-17 08:44:13,472 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 740bd1e2
>2012-12-17 08:44:13,472 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:44:21,609 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [6c95e244] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:44:22,244 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:44:22,432 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [107aac36] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:44:22,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4ac66000
>2012-12-17 08:44:22,486 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 291a661b
>2012-12-17 08:44:22,505 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [107aac36] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:22,690 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, SpmStopVDSCommand, log id: 291a661b
>2012-12-17 08:44:22,703 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, ResetIrsVDSCommand, log id: 4ac66000
>2012-12-17 08:44:22,715 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4d71b561
>2012-12-17 08:44:22,715 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [107aac36] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:22,715 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4d71b561
>2012-12-17 08:44:31,880 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:44:32,762 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [24452d4e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:44:32,775 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [24452d4e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4635
>2012-12-17 08:44:32,776 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4635, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 480be024
>2012-12-17 08:44:32,798 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] spmStart polling started: taskId = efa79c84-0c93-4e9e-9125-3b9b4731c98e
>2012-12-17 08:44:33,820 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] spmStart polling ended: taskId = efa79c84-0c93-4e9e-9125-3b9b4731c98e task status = finished
>2012-12-17 08:44:33,839 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] spmStart polling ended, spm status: SPM
>2012-12-17 08:44:33,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=efa79c84-0c93-4e9e-9125-3b9b4731c98e), log id: 4d33545f
>2012-12-17 08:44:33,858 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] FINISH, HSMClearTaskVDSCommand, log id: 4d33545f
>2012-12-17 08:44:33,858 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@64e85091, log id: 480be024
>2012-12-17 08:44:33,870 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [24452d4e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:44:33,934 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [55d82264] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:44:33,934 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: cac3416
>2012-12-17 08:44:33,963 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [24452d4e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:44:34,147 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [7ad32f3e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:44:34,206 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [7ad32f3e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4da16536
>2012-12-17 08:44:34,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [7ad32f3e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6e8a8494
>2012-12-17 08:44:34,227 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [7ad32f3e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:34,367 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [7ad32f3e] FINISH, SpmStopVDSCommand, log id: 6e8a8494
>2012-12-17 08:44:34,382 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [7ad32f3e] FINISH, ResetIrsVDSCommand, log id: 4da16536
>2012-12-17 08:44:34,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [7ad32f3e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3de8a623
>2012-12-17 08:44:34,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7ad32f3e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:34,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [7ad32f3e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3de8a623
>2012-12-17 08:44:34,476 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:44:34,497 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4636
>2012-12-17 08:44:34,499 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4636, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 188edafe
>2012-12-17 08:44:34,521 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling started: taskId = 9fdad1fc-6c4c-4e13-a8d5-4dccc5354918
>2012-12-17 08:44:35,543 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling ended: taskId = 9fdad1fc-6c4c-4e13-a8d5-4dccc5354918 task status = finished
>2012-12-17 08:44:35,561 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling ended, spm status: SPM
>2012-12-17 08:44:35,562 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9fdad1fc-6c4c-4e13-a8d5-4dccc5354918), log id: 4833c79c
>2012-12-17 08:44:35,581 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, HSMClearTaskVDSCommand, log id: 4833c79c
>2012-12-17 08:44:35,581 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3f75f7cc, log id: 188edafe
>2012-12-17 08:44:36,016 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:44:36,349 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:36,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: dd9a1c
>2012-12-17 08:44:36,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: dd9a1c
>2012-12-17 08:44:36,375 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: cac3416
>2012-12-17 08:44:36,375 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [55d82264] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:44:36,375 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 54b0a4b5
>2012-12-17 08:44:36,375 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:36,377 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 510bfc78
>2012-12-17 08:44:36,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 510bfc78
>2012-12-17 08:44:36,395 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 54b0a4b5
>2012-12-17 08:44:36,395 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [55d82264] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:44:42,052 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [48f6cd0b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:44:44,509 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [51589818] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:44:44,641 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [3f9576fc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:44:44,704 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 45a15c4f
>2012-12-17 08:44:44,706 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2bc46a7d
>2012-12-17 08:44:44,725 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:45,540 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] FINISH, SpmStopVDSCommand, log id: 2bc46a7d
>2012-12-17 08:44:45,964 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] FINISH, ResetIrsVDSCommand, log id: 45a15c4f
>2012-12-17 08:44:45,975 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 47f9392a
>2012-12-17 08:44:45,975 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:45,975 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 47f9392a
>2012-12-17 08:44:52,230 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:44:56,321 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [f91baee] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:44:56,343 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [f91baee] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4637
>2012-12-17 08:44:56,345 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [f91baee] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4637, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 261f56a3
>2012-12-17 08:44:56,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [f91baee] spmStart polling started: taskId = 4d8337f7-37e5-4efa-9fa4-ce887e8f4525
>2012-12-17 08:44:57,390 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [f91baee] spmStart polling ended: taskId = 4d8337f7-37e5-4efa-9fa4-ce887e8f4525 task status = finished
>2012-12-17 08:44:57,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [f91baee] spmStart polling ended, spm status: SPM
>2012-12-17 08:44:57,411 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [f91baee] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4d8337f7-37e5-4efa-9fa4-ce887e8f4525), log id: 7e1e5c8d
>2012-12-17 08:44:57,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [f91baee] FINISH, HSMClearTaskVDSCommand, log id: 7e1e5c8d
>2012-12-17 08:44:57,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [f91baee] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7b4e6daa, log id: 261f56a3
>2012-12-17 08:44:57,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [f91baee] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:44:57,512 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:44:57,512 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6dc881f0
>2012-12-17 08:44:57,531 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [f91baee] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:44:57,692 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [3d206f3c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:44:57,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 303f287c
>2012-12-17 08:44:57,746 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7499f28
>2012-12-17 08:44:57,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:58,107 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, SpmStopVDSCommand, log id: 7499f28
>2012-12-17 08:44:58,135 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, ResetIrsVDSCommand, log id: 303f287c
>2012-12-17 08:44:58,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3e28a95
>2012-12-17 08:44:58,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3d206f3c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:58,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3e28a95
>2012-12-17 08:44:58,200 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:44:58,221 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4638
>2012-12-17 08:44:58,222 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4638, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7c49fe60
>2012-12-17 08:44:58,246 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling started: taskId = 0e4bcbb4-c94b-474d-9857-527c70a4e7c3
>2012-12-17 08:44:59,269 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended: taskId = 0e4bcbb4-c94b-474d-9857-527c70a4e7c3 task status = finished
>2012-12-17 08:44:59,288 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended, spm status: SPM
>2012-12-17 08:44:59,289 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0e4bcbb4-c94b-474d-9857-527c70a4e7c3), log id: 631218b6
>2012-12-17 08:44:59,308 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMClearTaskVDSCommand, log id: 631218b6
>2012-12-17 08:44:59,308 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3914878a, log id: 7c49fe60
>2012-12-17 08:44:59,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:44:59,333 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:59,335 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 703581d5
>2012-12-17 08:44:59,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 703581d5
>2012-12-17 08:44:59,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6dc881f0
>2012-12-17 08:44:59,354 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:44:59,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2899f8e0
>2012-12-17 08:44:59,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:44:59,356 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 345d0a0e
>2012-12-17 08:44:59,374 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 345d0a0e
>2012-12-17 08:44:59,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2899f8e0
>2012-12-17 08:44:59,374 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:45:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [1ab078c7] Checking autorecoverable hosts
>2012-12-17 08:45:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [1ab078c7] Autorecovering 0 hosts
>2012-12-17 08:45:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [1ab078c7] Checking autorecoverable hosts done
>2012-12-17 08:45:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [1ab078c7] Autorecovering storage domains is disabled, skipping
>2012-12-17 08:45:02,462 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:45:08,253 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [b1109c1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:45:08,388 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [58b8166b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:45:08,447 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 14e1fa40
>2012-12-17 08:45:08,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 21b88f07
>2012-12-17 08:45:08,469 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:08,690 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] FINISH, SpmStopVDSCommand, log id: 21b88f07
>2012-12-17 08:45:08,737 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] FINISH, ResetIrsVDSCommand, log id: 14e1fa40
>2012-12-17 08:45:08,748 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3d0e6b55
>2012-12-17 08:45:08,748 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [58b8166b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:08,748 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3d0e6b55
>2012-12-17 08:45:12,704 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:45:18,802 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [5863fdac] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:45:18,820 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [5863fdac] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4639
>2012-12-17 08:45:18,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4639, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 32d8080c
>2012-12-17 08:45:18,843 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] spmStart polling started: taskId = e46086e4-41fb-4dd4-9fd5-6ca8e2b86ebd
>2012-12-17 08:45:19,869 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] spmStart polling ended: taskId = e46086e4-41fb-4dd4-9fd5-6ca8e2b86ebd task status = finished
>2012-12-17 08:45:19,889 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] spmStart polling ended, spm status: SPM
>2012-12-17 08:45:19,890 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e46086e4-41fb-4dd4-9fd5-6ca8e2b86ebd), log id: 7b831ec9
>2012-12-17 08:45:19,910 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] FINISH, HSMClearTaskVDSCommand, log id: 7b831ec9
>2012-12-17 08:45:19,911 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d5a5d78, log id: 32d8080c
>2012-12-17 08:45:19,924 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [5863fdac] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:45:19,963 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:45:19,963 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 74d05267
>2012-12-17 08:45:19,981 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [5863fdac] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:45:20,079 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:45:20,128 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3c4f4da5
>2012-12-17 08:45:20,130 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1125beb1
>2012-12-17 08:45:20,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:20,310 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, SpmStopVDSCommand, log id: 1125beb1
>2012-12-17 08:45:21,136 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, ResetIrsVDSCommand, log id: 3c4f4da5
>2012-12-17 08:45:21,149 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49ba87df
>2012-12-17 08:45:21,149 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:21,149 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49ba87df
>2012-12-17 08:45:21,343 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:45:21,364 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4640
>2012-12-17 08:45:21,365 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4640, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 90c5b1b
>2012-12-17 08:45:21,387 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling started: taskId = c012a391-1710-40e3-8b67-41dec16418d5
>2012-12-17 08:45:22,410 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended: taskId = c012a391-1710-40e3-8b67-41dec16418d5 task status = finished
>2012-12-17 08:45:22,428 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended, spm status: SPM
>2012-12-17 08:45:22,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c012a391-1710-40e3-8b67-41dec16418d5), log id: 78a5ee45
>2012-12-17 08:45:22,448 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMClearTaskVDSCommand, log id: 78a5ee45
>2012-12-17 08:45:22,448 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@55be31e4, log id: 90c5b1b
>2012-12-17 08:45:22,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:45:22,475 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:22,479 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 67eb7667
>2012-12-17 08:45:22,498 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 67eb7667
>2012-12-17 08:45:22,498 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 74d05267
>2012-12-17 08:45:22,498 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:45:22,498 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 25432948
>2012-12-17 08:45:22,499 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:22,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2d78401e
>2012-12-17 08:45:22,519 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2d78401e
>2012-12-17 08:45:22,519 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 25432948
>2012-12-17 08:45:22,519 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:45:22,893 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [717cca29] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:45:31,382 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [65a08bc7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:45:31,548 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [786fe4b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:45:31,653 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 72c567eb
>2012-12-17 08:45:31,655 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 15aa1c0
>2012-12-17 08:45:31,677 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:31,835 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, SpmStopVDSCommand, log id: 15aa1c0
>2012-12-17 08:45:31,876 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, ResetIrsVDSCommand, log id: 72c567eb
>2012-12-17 08:45:31,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1118a1cc
>2012-12-17 08:45:31,888 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [786fe4b5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:31,888 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1118a1cc
>2012-12-17 08:45:33,120 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [68b5cbc1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:45:41,939 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:45:41,945 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4641
>2012-12-17 08:45:41,947 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4641, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5928f331
>2012-12-17 08:45:41,970 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] spmStart polling started: taskId = 2b6bcbb5-e205-4e06-8bb2-1bf009f5da27
>2012-12-17 08:45:42,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] spmStart polling ended: taskId = 2b6bcbb5-e205-4e06-8bb2-1bf009f5da27 task status = finished
>2012-12-17 08:45:43,011 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] spmStart polling ended, spm status: SPM
>2012-12-17 08:45:43,012 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2b6bcbb5-e205-4e06-8bb2-1bf009f5da27), log id: 6339ab12
>2012-12-17 08:45:43,032 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, HSMClearTaskVDSCommand, log id: 6339ab12
>2012-12-17 08:45:43,032 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5f0e5da2, log id: 5928f331
>2012-12-17 08:45:43,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:45:43,082 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:45:43,083 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1dd5371
>2012-12-17 08:45:43,101 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:45:43,182 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-49) [536655ff] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:45:43,254 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [536655ff] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6ab7477a
>2012-12-17 08:45:43,255 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [536655ff] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: e4789f1
>2012-12-17 08:45:43,283 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [536655ff] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:43,360 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [bb53630] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:45:43,446 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [536655ff] FINISH, SpmStopVDSCommand, log id: e4789f1
>2012-12-17 08:45:43,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [536655ff] FINISH, ResetIrsVDSCommand, log id: 6ab7477a
>2012-12-17 08:45:43,495 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [536655ff] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4d0df750
>2012-12-17 08:45:43,495 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [536655ff] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:43,495 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [536655ff] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4d0df750
>2012-12-17 08:45:43,715 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:45:43,753 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4642
>2012-12-17 08:45:43,755 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4642, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6f0027a6
>2012-12-17 08:45:43,777 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling started: taskId = 215685ad-0f84-4099-8b04-4885cfe519cb
>2012-12-17 08:45:44,800 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling ended: taskId = 215685ad-0f84-4099-8b04-4885cfe519cb task status = finished
>2012-12-17 08:45:44,818 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling ended, spm status: SPM
>2012-12-17 08:45:44,820 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=215685ad-0f84-4099-8b04-4885cfe519cb), log id: 366a31d0
>2012-12-17 08:45:44,838 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMClearTaskVDSCommand, log id: 366a31d0
>2012-12-17 08:45:44,838 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@75ec6a98, log id: 6f0027a6
>2012-12-17 08:45:44,849 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:45:44,860 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:44,862 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5385f790
>2012-12-17 08:45:44,882 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5385f790
>2012-12-17 08:45:44,882 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1dd5371
>2012-12-17 08:45:44,882 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:45:44,882 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 789137f1
>2012-12-17 08:45:44,882 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:44,883 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:45:44,883 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7d0f43ab
>2012-12-17 08:45:44,884 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: aefec30
>2012-12-17 08:45:44,902 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: aefec30
>2012-12-17 08:45:44,902 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 789137f1
>2012-12-17 08:45:44,902 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:45:44,902 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:44,903 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2616771b
>2012-12-17 08:45:44,922 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2616771b
>2012-12-17 08:45:44,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7d0f43ab
>2012-12-17 08:45:44,922 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:45:44,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: ce5a126
>2012-12-17 08:45:44,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:44,923 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 10572dbf
>2012-12-17 08:45:44,942 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 10572dbf
>2012-12-17 08:45:44,942 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: ce5a126
>2012-12-17 08:45:44,942 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:45:53,761 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [41698d74] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:45:54,094 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [6cc0b445] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:45:54,166 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7dd34308
>2012-12-17 08:45:54,168 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 41de21e1
>2012-12-17 08:45:54,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:54,364 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, SpmStopVDSCommand, log id: 41de21e1
>2012-12-17 08:45:54,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, ResetIrsVDSCommand, log id: 7dd34308
>2012-12-17 08:45:54,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 18dcc6ea
>2012-12-17 08:45:54,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [6cc0b445] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:45:54,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 18dcc6ea
>2012-12-17 08:46:03,731 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [34599a45] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:46:04,445 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [56986c52] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:46:04,464 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [56986c52] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4643
>2012-12-17 08:46:04,465 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [56986c52] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4643, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 764f82e5
>2012-12-17 08:46:04,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [56986c52] spmStart polling started: taskId = 3597ab48-d31a-40cb-ae91-fb17bc14740e
>2012-12-17 08:46:05,511 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [56986c52] spmStart polling ended: taskId = 3597ab48-d31a-40cb-ae91-fb17bc14740e task status = finished
>2012-12-17 08:46:05,529 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [56986c52] spmStart polling ended, spm status: SPM
>2012-12-17 08:46:05,531 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [56986c52] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3597ab48-d31a-40cb-ae91-fb17bc14740e), log id: c308e8c
>2012-12-17 08:46:05,549 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [56986c52] FINISH, HSMClearTaskVDSCommand, log id: c308e8c
>2012-12-17 08:46:05,549 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [56986c52] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@47de8f94, log id: 764f82e5
>2012-12-17 08:46:05,963 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [56986c52] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:46:06,106 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:46:06,106 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3d0449c
>2012-12-17 08:46:06,116 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [56986c52] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:46:06,316 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [32ec1ec3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:46:06,360 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7b00e96b
>2012-12-17 08:46:06,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26ec42f0
>2012-12-17 08:46:06,382 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:06,520 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] FINISH, SpmStopVDSCommand, log id: 26ec42f0
>2012-12-17 08:46:06,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] FINISH, ResetIrsVDSCommand, log id: 7b00e96b
>2012-12-17 08:46:06,557 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 25d0f66c
>2012-12-17 08:46:06,557 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [32ec1ec3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:06,557 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 25d0f66c
>2012-12-17 08:46:06,645 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:46:06,653 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4644
>2012-12-17 08:46:06,654 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4644, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 485ba2b2
>2012-12-17 08:46:06,676 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling started: taskId = 684d6c9d-2b10-4131-a407-1ab0bb9f81df
>2012-12-17 08:46:07,699 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended: taskId = 684d6c9d-2b10-4131-a407-1ab0bb9f81df task status = finished
>2012-12-17 08:46:07,717 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended, spm status: SPM
>2012-12-17 08:46:07,718 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=684d6c9d-2b10-4131-a407-1ab0bb9f81df), log id: e57fd24
>2012-12-17 08:46:07,737 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMClearTaskVDSCommand, log id: e57fd24
>2012-12-17 08:46:07,737 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@32ac4dd7, log id: 485ba2b2
>2012-12-17 08:46:07,749 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:46:07,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:07,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2f5ba3b9
>2012-12-17 08:46:07,806 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2f5ba3b9
>2012-12-17 08:46:07,806 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3d0449c
>2012-12-17 08:46:07,806 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:46:07,806 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 68c42198
>2012-12-17 08:46:07,806 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:07,808 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5e56325c
>2012-12-17 08:46:07,826 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5e56325c
>2012-12-17 08:46:07,826 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 68c42198
>2012-12-17 08:46:07,826 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:46:13,981 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [30867547] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:46:16,679 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7f13ba47] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:46:16,791 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [1a0f716] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:46:16,900 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 39ad0a19
>2012-12-17 08:46:16,902 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6f001e6e
>2012-12-17 08:46:16,922 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:17,069 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] FINISH, SpmStopVDSCommand, log id: 6f001e6e
>2012-12-17 08:46:17,083 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] FINISH, ResetIrsVDSCommand, log id: 39ad0a19
>2012-12-17 08:46:17,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1642bcba
>2012-12-17 08:46:17,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1a0f716] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:17,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1642bcba
>2012-12-17 08:46:24,173 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:46:27,193 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [27478f27] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:46:27,214 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [27478f27] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4645
>2012-12-17 08:46:27,216 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [27478f27] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4645, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6eddfb7c
>2012-12-17 08:46:27,237 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [27478f27] spmStart polling started: taskId = 0d2dc974-8736-4330-82bb-fa9488b08200
>2012-12-17 08:46:28,257 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [27478f27] spmStart polling ended: taskId = 0d2dc974-8736-4330-82bb-fa9488b08200 task status = finished
>2012-12-17 08:46:28,276 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [27478f27] spmStart polling ended, spm status: SPM
>2012-12-17 08:46:28,277 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [27478f27] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0d2dc974-8736-4330-82bb-fa9488b08200), log id: 539b4a27
>2012-12-17 08:46:28,294 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [27478f27] FINISH, HSMClearTaskVDSCommand, log id: 539b4a27
>2012-12-17 08:46:28,295 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [27478f27] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78217152, log id: 6eddfb7c
>2012-12-17 08:46:28,321 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [27478f27] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:46:28,361 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [7b0d0943] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:46:28,362 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: f5846ee
>2012-12-17 08:46:28,388 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [27478f27] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:46:28,490 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [d00852e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:46:28,565 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [d00852e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3d6cf6db
>2012-12-17 08:46:28,567 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [d00852e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 11ec7931
>2012-12-17 08:46:28,586 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [d00852e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:28,949 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [d00852e] FINISH, SpmStopVDSCommand, log id: 11ec7931
>2012-12-17 08:46:28,957 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [d00852e] FINISH, ResetIrsVDSCommand, log id: 3d6cf6db
>2012-12-17 08:46:28,968 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [d00852e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e8d1dec
>2012-12-17 08:46:28,968 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [d00852e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:28,968 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [d00852e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e8d1dec
>2012-12-17 08:46:29,058 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:46:29,066 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4646
>2012-12-17 08:46:29,068 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4646, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3f7c759a
>2012-12-17 08:46:29,091 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling started: taskId = 2711d0a9-575f-4601-a3e0-2c7bf4f3c0f0
>2012-12-17 08:46:30,113 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling ended: taskId = 2711d0a9-575f-4601-a3e0-2c7bf4f3c0f0 task status = finished
>2012-12-17 08:46:30,131 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling ended, spm status: SPM
>2012-12-17 08:46:30,132 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2711d0a9-575f-4601-a3e0-2c7bf4f3c0f0), log id: 2a085f3c
>2012-12-17 08:46:30,150 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, HSMClearTaskVDSCommand, log id: 2a085f3c
>2012-12-17 08:46:30,150 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@ba471b3, log id: 3f7c759a
>2012-12-17 08:46:30,162 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:46:30,189 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:30,193 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 737d76b1
>2012-12-17 08:46:30,212 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 737d76b1
>2012-12-17 08:46:30,212 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: f5846ee
>2012-12-17 08:46:30,212 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [7b0d0943] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:46:30,212 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7d27d067
>2012-12-17 08:46:30,212 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:30,213 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7063efd7
>2012-12-17 08:46:30,237 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7063efd7
>2012-12-17 08:46:30,237 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7d27d067
>2012-12-17 08:46:30,237 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [7b0d0943] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:46:34,364 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:46:39,119 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [5970076c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:46:39,423 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-33) [75b7a65b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:46:39,539 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 676a499c
>2012-12-17 08:46:39,541 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2e0523b8
>2012-12-17 08:46:39,561 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:39,854 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SpmStopVDSCommand, log id: 2e0523b8
>2012-12-17 08:46:39,869 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, ResetIrsVDSCommand, log id: 676a499c
>2012-12-17 08:46:39,880 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ef39ce8
>2012-12-17 08:46:39,880 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:39,880 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ef39ce8
>2012-12-17 08:46:44,572 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [58c21a2d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:46:49,935 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b5bb621] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:46:50,004 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b5bb621] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4647
>2012-12-17 08:46:50,005 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4647, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62c94fb2
>2012-12-17 08:46:50,028 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] spmStart polling started: taskId = 74bbd10d-2ace-4bd1-bd5e-b5f5b4ec8691
>2012-12-17 08:46:51,055 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] spmStart polling ended: taskId = 74bbd10d-2ace-4bd1-bd5e-b5f5b4ec8691 task status = finished
>2012-12-17 08:46:51,074 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] spmStart polling ended, spm status: SPM
>2012-12-17 08:46:51,076 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=74bbd10d-2ace-4bd1-bd5e-b5f5b4ec8691), log id: 7816b8f1
>2012-12-17 08:46:51,095 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] FINISH, HSMClearTaskVDSCommand, log id: 7816b8f1
>2012-12-17 08:46:51,095 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1bc813cf, log id: 62c94fb2
>2012-12-17 08:46:51,113 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b5bb621] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:46:51,148 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [50f32e72] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:46:51,149 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 764e3aef
>2012-12-17 08:46:51,213 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b5bb621] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:46:51,385 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-51) [71efd086] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:46:51,445 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [71efd086] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 69056b4
>2012-12-17 08:46:51,447 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [71efd086] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4440f193
>2012-12-17 08:46:51,467 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [71efd086] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:51,703 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [71efd086] FINISH, SpmStopVDSCommand, log id: 4440f193
>2012-12-17 08:46:51,733 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [71efd086] FINISH, ResetIrsVDSCommand, log id: 69056b4
>2012-12-17 08:46:51,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [71efd086] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 21dbfb83
>2012-12-17 08:46:51,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:51,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [71efd086] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 21dbfb83
>2012-12-17 08:46:51,790 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:46:51,805 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4648
>2012-12-17 08:46:51,806 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4648, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5737c9c
>2012-12-17 08:46:51,828 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] spmStart polling started: taskId = e7ac015c-3dbe-49cb-a066-159879c3d1d5
>2012-12-17 08:46:52,849 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] spmStart polling ended: taskId = e7ac015c-3dbe-49cb-a066-159879c3d1d5 task status = finished
>2012-12-17 08:46:52,867 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] spmStart polling ended, spm status: SPM
>2012-12-17 08:46:52,868 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e7ac015c-3dbe-49cb-a066-159879c3d1d5), log id: 56a8d7eb
>2012-12-17 08:46:52,887 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, HSMClearTaskVDSCommand, log id: 56a8d7eb
>2012-12-17 08:46:52,887 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@66f7b1bf, log id: 5737c9c
>2012-12-17 08:46:52,897 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:46:52,907 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:52,912 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 373b2aac
>2012-12-17 08:46:52,931 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 373b2aac
>2012-12-17 08:46:52,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 764e3aef
>2012-12-17 08:46:52,931 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [50f32e72] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:46:52,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: d6e9e88
>2012-12-17 08:46:52,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:46:52,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4b8c9512
>2012-12-17 08:46:52,952 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b8c9512
>2012-12-17 08:46:52,952 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: d6e9e88
>2012-12-17 08:46:52,952 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [50f32e72] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:46:54,747 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:47:01,828 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2649fcd5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:47:02,042 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-62) [3e0f12eb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:47:02,110 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5fc734e1
>2012-12-17 08:47:02,112 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 64e79e64
>2012-12-17 08:47:02,132 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:02,293 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, SpmStopVDSCommand, log id: 64e79e64
>2012-12-17 08:47:02,379 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, ResetIrsVDSCommand, log id: 5fc734e1
>2012-12-17 08:47:02,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 53a984a1
>2012-12-17 08:47:02,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:02,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 53a984a1
>2012-12-17 08:47:04,917 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [495d106] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:47:12,467 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:47:12,478 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4649
>2012-12-17 08:47:12,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4649, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 8f8fb4c
>2012-12-17 08:47:12,501 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] spmStart polling started: taskId = 19c08f48-cb51-4489-b783-5fb3c6afc42f
>2012-12-17 08:47:13,522 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] spmStart polling ended: taskId = 19c08f48-cb51-4489-b783-5fb3c6afc42f task status = finished
>2012-12-17 08:47:13,540 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] spmStart polling ended, spm status: SPM
>2012-12-17 08:47:13,541 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=19c08f48-cb51-4489-b783-5fb3c6afc42f), log id: 5245350d
>2012-12-17 08:47:13,560 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, HSMClearTaskVDSCommand, log id: 5245350d
>2012-12-17 08:47:13,560 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3d9e12ad, log id: 8f8fb4c
>2012-12-17 08:47:13,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:47:13,707 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-37) [783fb511] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:47:13,707 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: b71578a
>2012-12-17 08:47:13,729 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:47:14,039 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [726a9e10] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:47:14,114 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [726a9e10] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17a78ee7
>2012-12-17 08:47:14,155 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [726a9e10] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4d304603
>2012-12-17 08:47:14,176 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [726a9e10] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:14,328 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [726a9e10] FINISH, SpmStopVDSCommand, log id: 4d304603
>2012-12-17 08:47:14,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [726a9e10] FINISH, ResetIrsVDSCommand, log id: 17a78ee7
>2012-12-17 08:47:14,367 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [726a9e10] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40dcfb98
>2012-12-17 08:47:14,367 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [726a9e10] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:14,367 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [726a9e10] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40dcfb98
>2012-12-17 08:47:14,464 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:47:14,490 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4650
>2012-12-17 08:47:14,491 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4650, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 139a354b
>2012-12-17 08:47:14,513 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [783fb511] spmStart polling started: taskId = f7af5754-eb53-4cb3-aa4e-c72e6c252366
>2012-12-17 08:47:16,079 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [783fb511] spmStart polling ended: taskId = f7af5754-eb53-4cb3-aa4e-c72e6c252366 task status = finished
>2012-12-17 08:47:16,099 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [783fb511] spmStart polling ended, spm status: SPM
>2012-12-17 08:47:16,101 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f7af5754-eb53-4cb3-aa4e-c72e6c252366), log id: 77d79ef0
>2012-12-17 08:47:16,120 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, HSMClearTaskVDSCommand, log id: 77d79ef0
>2012-12-17 08:47:16,120 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6d03dc07, log id: 139a354b
>2012-12-17 08:47:16,269 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:47:16,331 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:16,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3e07d210
>2012-12-17 08:47:16,352 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3e07d210
>2012-12-17 08:47:16,353 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: b71578a
>2012-12-17 08:47:16,353 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-37) [783fb511] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:47:16,353 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 73703840
>2012-12-17 08:47:16,353 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:16,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2a592ce0
>2012-12-17 08:47:16,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2a592ce0
>2012-12-17 08:47:16,373 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 73703840
>2012-12-17 08:47:16,373 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-37) [783fb511] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:47:24,508 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:47:24,620 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-35) [388f9500] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:47:24,772 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [388f9500] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 532c6f09
>2012-12-17 08:47:24,774 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [388f9500] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 79b09590
>2012-12-17 08:47:24,794 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [388f9500] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:24,954 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [388f9500] FINISH, SpmStopVDSCommand, log id: 79b09590
>2012-12-17 08:47:24,972 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [388f9500] FINISH, ResetIrsVDSCommand, log id: 532c6f09
>2012-12-17 08:47:24,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [388f9500] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4baa33a9
>2012-12-17 08:47:24,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [388f9500] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:24,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [388f9500] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4baa33a9
>2012-12-17 08:47:25,263 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:47:35,036 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [4595e4ad] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:47:35,048 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [4595e4ad] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4651
>2012-12-17 08:47:35,049 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4651, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2bc77fe5
>2012-12-17 08:47:35,073 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] spmStart polling started: taskId = 7753b6d7-716e-4b34-912f-ff51b3790a76
>2012-12-17 08:47:36,096 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] spmStart polling ended: taskId = 7753b6d7-716e-4b34-912f-ff51b3790a76 task status = finished
>2012-12-17 08:47:36,116 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] spmStart polling ended, spm status: SPM
>2012-12-17 08:47:36,118 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7753b6d7-716e-4b34-912f-ff51b3790a76), log id: 4a8b2a07
>2012-12-17 08:47:36,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] FINISH, HSMClearTaskVDSCommand, log id: 4a8b2a07
>2012-12-17 08:47:36,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@767457cc, log id: 2bc77fe5
>2012-12-17 08:47:36,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [4595e4ad] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:47:36,177 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-34) [4c63d330] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:47:36,177 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 34332a3a
>2012-12-17 08:47:36,193 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [4595e4ad] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:47:36,300 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [2de700da] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:47:36,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [2de700da] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 71442728
>2012-12-17 08:47:36,402 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [2de700da] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6a53a09b
>2012-12-17 08:47:36,422 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [2de700da] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:36,548 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [2de700da] FINISH, SpmStopVDSCommand, log id: 6a53a09b
>2012-12-17 08:47:36,571 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [2de700da] FINISH, ResetIrsVDSCommand, log id: 71442728
>2012-12-17 08:47:36,582 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [2de700da] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7e4fbec5
>2012-12-17 08:47:36,582 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [2de700da] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:36,582 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [2de700da] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7e4fbec5
>2012-12-17 08:47:36,649 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [4c63d330] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:47:36,658 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [4c63d330] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4652
>2012-12-17 08:47:36,659 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4652, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 78eb65fd
>2012-12-17 08:47:36,682 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] spmStart polling started: taskId = a7273340-85a6-4978-b25f-4dab207a2637
>2012-12-17 08:47:37,704 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] spmStart polling ended: taskId = a7273340-85a6-4978-b25f-4dab207a2637 task status = finished
>2012-12-17 08:47:37,722 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] spmStart polling ended, spm status: SPM
>2012-12-17 08:47:37,723 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a7273340-85a6-4978-b25f-4dab207a2637), log id: 22ab54c1
>2012-12-17 08:47:37,741 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] FINISH, HSMClearTaskVDSCommand, log id: 22ab54c1
>2012-12-17 08:47:37,742 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1e495caf, log id: 78eb65fd
>2012-12-17 08:47:37,751 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [4c63d330] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:47:37,776 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:37,777 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4155716d
>2012-12-17 08:47:37,797 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4155716d
>2012-12-17 08:47:37,797 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 34332a3a
>2012-12-17 08:47:37,797 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-34) [4c63d330] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:47:37,797 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5089cc9d
>2012-12-17 08:47:37,797 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:37,799 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4d360188
>2012-12-17 08:47:37,817 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4d360188
>2012-12-17 08:47:37,817 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5089cc9d
>2012-12-17 08:47:37,817 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-34) [4c63d330] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:47:46,719 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [717cca29] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:47:46,756 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:47:46,894 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [1bff0c70] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:47:46,942 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 24681b6
>2012-12-17 08:47:46,943 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 444dddc1
>2012-12-17 08:47:46,964 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:47,115 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, SpmStopVDSCommand, log id: 444dddc1
>2012-12-17 08:47:47,138 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, ResetIrsVDSCommand, log id: 24681b6
>2012-12-17 08:47:47,150 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 39131659
>2012-12-17 08:47:47,150 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1bff0c70] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:47,150 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 39131659
>2012-12-17 08:47:57,071 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:47:57,297 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [68b5cbc1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:47:57,321 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [68b5cbc1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4653
>2012-12-17 08:47:57,322 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4653, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7be4d3a
>2012-12-17 08:47:57,348 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] spmStart polling started: taskId = 72f19c36-523d-4a5c-b296-989215321f55
>2012-12-17 08:47:58,371 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] spmStart polling ended: taskId = 72f19c36-523d-4a5c-b296-989215321f55 task status = finished
>2012-12-17 08:47:58,389 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] spmStart polling ended, spm status: SPM
>2012-12-17 08:47:58,390 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=72f19c36-523d-4a5c-b296-989215321f55), log id: 3e05899a
>2012-12-17 08:47:58,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] FINISH, HSMClearTaskVDSCommand, log id: 3e05899a
>2012-12-17 08:47:58,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d8cfd45, log id: 7be4d3a
>2012-12-17 08:47:58,430 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [68b5cbc1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:47:58,464 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [40a5fc59] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:47:58,464 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5e9abb6e
>2012-12-17 08:47:58,480 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [68b5cbc1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:47:58,603 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [446aaffb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:47:58,704 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 86ccb70
>2012-12-17 08:47:58,706 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 40b48f80
>2012-12-17 08:47:58,726 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:59,190 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, SpmStopVDSCommand, log id: 40b48f80
>2012-12-17 08:47:59,207 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, ResetIrsVDSCommand, log id: 86ccb70
>2012-12-17 08:47:59,218 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a09f11d
>2012-12-17 08:47:59,218 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:47:59,218 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a09f11d
>2012-12-17 08:47:59,329 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [40a5fc59] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:47:59,360 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [40a5fc59] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4654
>2012-12-17 08:47:59,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4654, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 401e1417
>2012-12-17 08:47:59,387 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] spmStart polling started: taskId = ca85646b-67f1-4f48-b074-94f4b96c1fa3
>2012-12-17 08:48:00,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] spmStart polling ended: taskId = ca85646b-67f1-4f48-b074-94f4b96c1fa3 task status = finished
>2012-12-17 08:48:00,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] spmStart polling ended, spm status: SPM
>2012-12-17 08:48:00,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ca85646b-67f1-4f48-b074-94f4b96c1fa3), log id: 251af9e0
>2012-12-17 08:48:00,447 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, HSMClearTaskVDSCommand, log id: 251af9e0
>2012-12-17 08:48:00,447 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e0af428, log id: 401e1417
>2012-12-17 08:48:01,129 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [40a5fc59] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:48:01,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:01,293 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 19169891
>2012-12-17 08:48:01,315 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 19169891
>2012-12-17 08:48:01,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5e9abb6e
>2012-12-17 08:48:01,315 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [40a5fc59] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:48:01,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1a7a6475
>2012-12-17 08:48:01,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:01,316 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 15b44e60
>2012-12-17 08:48:01,339 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 15b44e60
>2012-12-17 08:48:01,339 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1a7a6475
>2012-12-17 08:48:01,339 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [40a5fc59] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:48:07,300 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [1ee6f70] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:48:09,397 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [381e4566] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:48:09,574 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [15ef2937] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:48:09,642 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 901e857
>2012-12-17 08:48:09,643 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 215a8047
>2012-12-17 08:48:09,663 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:09,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, SpmStopVDSCommand, log id: 215a8047
>2012-12-17 08:48:09,845 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, ResetIrsVDSCommand, log id: 901e857
>2012-12-17 08:48:09,856 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5d53f90c
>2012-12-17 08:48:09,856 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [15ef2937] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:09,856 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5d53f90c
>2012-12-17 08:48:17,550 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:48:19,897 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [58b8166b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:48:19,917 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [58b8166b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4655
>2012-12-17 08:48:19,919 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4655, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3ff68367
>2012-12-17 08:48:19,942 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] spmStart polling started: taskId = 2e129f9d-cc40-415f-a718-55759e3ec462
>2012-12-17 08:48:20,964 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] spmStart polling ended: taskId = 2e129f9d-cc40-415f-a718-55759e3ec462 task status = finished
>2012-12-17 08:48:20,983 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] spmStart polling ended, spm status: SPM
>2012-12-17 08:48:20,984 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2e129f9d-cc40-415f-a718-55759e3ec462), log id: 55ff7930
>2012-12-17 08:48:21,003 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] FINISH, HSMClearTaskVDSCommand, log id: 55ff7930
>2012-12-17 08:48:21,003 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@776d6a6b, log id: 3ff68367
>2012-12-17 08:48:21,262 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [58b8166b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:48:21,392 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-31) [107aac36] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:48:21,392 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4b74b3f3
>2012-12-17 08:48:21,423 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [58b8166b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:48:21,540 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [3969aa82] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:48:21,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5bd05288
>2012-12-17 08:48:21,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 452c700
>2012-12-17 08:48:21,620 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:21,781 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] FINISH, SpmStopVDSCommand, log id: 452c700
>2012-12-17 08:48:21,817 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] FINISH, ResetIrsVDSCommand, log id: 5bd05288
>2012-12-17 08:48:21,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 330f18fe
>2012-12-17 08:48:21,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3969aa82] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:21,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 330f18fe
>2012-12-17 08:48:21,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [107aac36] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:48:21,898 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [107aac36] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4656
>2012-12-17 08:48:21,899 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4656, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 117925bb
>2012-12-17 08:48:21,922 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [107aac36] spmStart polling started: taskId = 20fe5b72-2b7e-48f5-b8eb-4a3dff33a28c
>2012-12-17 08:48:22,946 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [107aac36] spmStart polling ended: taskId = 20fe5b72-2b7e-48f5-b8eb-4a3dff33a28c task status = finished
>2012-12-17 08:48:22,964 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [107aac36] spmStart polling ended, spm status: SPM
>2012-12-17 08:48:22,965 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=20fe5b72-2b7e-48f5-b8eb-4a3dff33a28c), log id: 202fcbe9
>2012-12-17 08:48:22,983 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, HSMClearTaskVDSCommand, log id: 202fcbe9
>2012-12-17 08:48:22,983 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@21f43e81, log id: 117925bb
>2012-12-17 08:48:23,012 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [107aac36] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:48:23,027 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:23,031 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 991be02
>2012-12-17 08:48:23,050 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 991be02
>2012-12-17 08:48:23,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4b74b3f3
>2012-12-17 08:48:23,050 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-31) [107aac36] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:48:23,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3f517a36
>2012-12-17 08:48:23,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:23,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2cdaacf7
>2012-12-17 08:48:23,070 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2cdaacf7
>2012-12-17 08:48:23,070 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f517a36
>2012-12-17 08:48:23,070 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-31) [107aac36] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:48:27,786 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:48:31,959 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7ad32f3e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:48:32,090 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [155e23b1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:48:32,136 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 75b19215
>2012-12-17 08:48:32,138 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7b6327d9
>2012-12-17 08:48:32,157 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:32,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, SpmStopVDSCommand, log id: 7b6327d9
>2012-12-17 08:48:32,392 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, ResetIrsVDSCommand, log id: 75b19215
>2012-12-17 08:48:32,402 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 621b05be
>2012-12-17 08:48:32,402 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:32,402 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 621b05be
>2012-12-17 08:48:37,978 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [30867547] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:48:42,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [6f019f85] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:48:42,474 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [6f019f85] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4657
>2012-12-17 08:48:42,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4657, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1ac81200
>2012-12-17 08:48:42,497 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] spmStart polling started: taskId = e0d4b294-cd04-4205-8f00-49241b117e8c
>2012-12-17 08:48:43,519 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] spmStart polling ended: taskId = e0d4b294-cd04-4205-8f00-49241b117e8c task status = finished
>2012-12-17 08:48:43,539 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] spmStart polling ended, spm status: SPM
>2012-12-17 08:48:43,540 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e0d4b294-cd04-4205-8f00-49241b117e8c), log id: 33cbaeef
>2012-12-17 08:48:43,560 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] FINISH, HSMClearTaskVDSCommand, log id: 33cbaeef
>2012-12-17 08:48:43,560 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@149e77a, log id: 1ac81200
>2012-12-17 08:48:43,575 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [6f019f85] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:48:43,634 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-32) [786fe4b5] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:48:43,634 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4fe30d05
>2012-12-17 08:48:43,665 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [6f019f85] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:48:43,782 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [27a8ed97] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:48:43,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 406ddfcc
>2012-12-17 08:48:43,834 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 34e0b9b
>2012-12-17 08:48:43,854 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:44,002 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, SpmStopVDSCommand, log id: 34e0b9b
>2012-12-17 08:48:44,015 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, ResetIrsVDSCommand, log id: 406ddfcc
>2012-12-17 08:48:44,026 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 279c62a1
>2012-12-17 08:48:44,026 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:44,026 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 279c62a1
>2012-12-17 08:48:44,100 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [786fe4b5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:48:44,127 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [786fe4b5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4658
>2012-12-17 08:48:44,129 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4658, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 234667e2
>2012-12-17 08:48:44,152 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] spmStart polling started: taskId = 61d5bb19-cf76-41ff-b7e0-4d7988b9af70
>2012-12-17 08:48:45,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] spmStart polling ended: taskId = 61d5bb19-cf76-41ff-b7e0-4d7988b9af70 task status = finished
>2012-12-17 08:48:45,194 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] spmStart polling ended, spm status: SPM
>2012-12-17 08:48:45,195 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=61d5bb19-cf76-41ff-b7e0-4d7988b9af70), log id: 38de7a4b
>2012-12-17 08:48:45,214 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, HSMClearTaskVDSCommand, log id: 38de7a4b
>2012-12-17 08:48:45,214 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@162eef18, log id: 234667e2
>2012-12-17 08:48:45,233 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [786fe4b5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:48:45,243 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:45,247 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1d784584
>2012-12-17 08:48:45,267 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1d784584
>2012-12-17 08:48:45,267 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4fe30d05
>2012-12-17 08:48:45,267 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-32) [786fe4b5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:48:45,267 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5e31b49a
>2012-12-17 08:48:45,267 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:45,268 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 12dc2726
>2012-12-17 08:48:45,287 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 12dc2726
>2012-12-17 08:48:45,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5e31b49a
>2012-12-17 08:48:45,287 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-32) [786fe4b5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:48:48,187 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [726a9e10] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:48:54,141 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [1ee6f70] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:48:54,309 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-18) [337c1daa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:48:54,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 62e977b
>2012-12-17 08:48:54,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3a19bfb
>2012-12-17 08:48:54,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:54,566 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] FINISH, SpmStopVDSCommand, log id: 3a19bfb
>2012-12-17 08:48:54,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] FINISH, ResetIrsVDSCommand, log id: 62e977b
>2012-12-17 08:48:54,592 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 19a23671
>2012-12-17 08:48:54,592 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [337c1daa] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:48:54,592 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 19a23671
>2012-12-17 08:48:58,379 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3eec3b83] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:49:04,662 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:49:04,710 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4659
>2012-12-17 08:49:04,712 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4659, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7f4c9760
>2012-12-17 08:49:04,735 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] spmStart polling started: taskId = 9c644c2d-5181-4c61-8326-67a3a343bd75
>2012-12-17 08:49:05,758 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] spmStart polling ended: taskId = 9c644c2d-5181-4c61-8326-67a3a343bd75 task status = finished
>2012-12-17 08:49:05,776 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] spmStart polling ended, spm status: SPM
>2012-12-17 08:49:05,777 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9c644c2d-5181-4c61-8326-67a3a343bd75), log id: 10a2e041
>2012-12-17 08:49:05,796 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, HSMClearTaskVDSCommand, log id: 10a2e041
>2012-12-17 08:49:05,796 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3530d3ef, log id: 7f4c9760
>2012-12-17 08:49:05,904 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:49:06,031 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [6cc0b445] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:49:06,031 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5046331
>2012-12-17 08:49:06,065 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:49:06,168 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [1a40d676] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:49:06,227 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [1a40d676] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 18a25764
>2012-12-17 08:49:06,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [1a40d676] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7efc42f4
>2012-12-17 08:49:06,248 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [1a40d676] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:06,369 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [1a40d676] FINISH, SpmStopVDSCommand, log id: 7efc42f4
>2012-12-17 08:49:06,379 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [1a40d676] FINISH, ResetIrsVDSCommand, log id: 18a25764
>2012-12-17 08:49:06,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [1a40d676] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 30332130
>2012-12-17 08:49:06,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [1a40d676] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:06,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [1a40d676] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 30332130
>2012-12-17 08:49:06,502 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [6cc0b445] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:49:06,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [6cc0b445] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4660
>2012-12-17 08:49:06,532 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4660, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 655dead0
>2012-12-17 08:49:06,554 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] spmStart polling started: taskId = 33254414-51e3-4227-b34e-675683f2940a
>2012-12-17 08:49:07,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] spmStart polling ended: taskId = 33254414-51e3-4227-b34e-675683f2940a task status = finished
>2012-12-17 08:49:07,594 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] spmStart polling ended, spm status: SPM
>2012-12-17 08:49:07,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=33254414-51e3-4227-b34e-675683f2940a), log id: 566949d
>2012-12-17 08:49:07,613 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, HSMClearTaskVDSCommand, log id: 566949d
>2012-12-17 08:49:07,614 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@11a6d230, log id: 655dead0
>2012-12-17 08:49:07,633 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [6cc0b445] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:49:07,681 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:07,682 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2ad7bd58
>2012-12-17 08:49:07,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2ad7bd58
>2012-12-17 08:49:07,701 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5046331
>2012-12-17 08:49:07,701 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [6cc0b445] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:49:07,701 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 331dbee
>2012-12-17 08:49:07,701 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:07,702 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 248391d8
>2012-12-17 08:49:07,720 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 248391d8
>2012-12-17 08:49:07,720 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 331dbee
>2012-12-17 08:49:07,720 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [6cc0b445] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:49:08,617 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3969aa82] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:49:16,539 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [28e17066] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:49:16,711 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [108329de] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:49:16,782 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [108329de] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 690dfb4
>2012-12-17 08:49:16,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [108329de] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6f561792
>2012-12-17 08:49:16,803 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [108329de] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:16,958 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [108329de] FINISH, SpmStopVDSCommand, log id: 6f561792
>2012-12-17 08:49:16,979 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [108329de] FINISH, ResetIrsVDSCommand, log id: 690dfb4
>2012-12-17 08:49:16,989 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [108329de] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 78504d03
>2012-12-17 08:49:16,989 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [108329de] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:16,989 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [108329de] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 78504d03
>2012-12-17 08:49:18,925 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:49:27,073 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5f1742db] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:49:27,102 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5f1742db] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4661
>2012-12-17 08:49:27,103 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4661, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4c2a2fd2
>2012-12-17 08:49:27,124 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] spmStart polling started: taskId = 3320270d-0705-4814-bcc9-5a8ee20a73cc
>2012-12-17 08:49:28,146 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] spmStart polling ended: taskId = 3320270d-0705-4814-bcc9-5a8ee20a73cc task status = finished
>2012-12-17 08:49:28,166 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] spmStart polling ended, spm status: SPM
>2012-12-17 08:49:28,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3320270d-0705-4814-bcc9-5a8ee20a73cc), log id: 5879adb7
>2012-12-17 08:49:28,186 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] FINISH, HSMClearTaskVDSCommand, log id: 5879adb7
>2012-12-17 08:49:28,186 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c7bf6c2, log id: 4c2a2fd2
>2012-12-17 08:49:28,219 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5f1742db] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:49:28,325 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [1f4324c3] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:49:28,325 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 583b9aa8
>2012-12-17 08:49:28,359 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5f1742db] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:49:28,560 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-4) [5bc809fe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:49:28,686 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 958fa36
>2012-12-17 08:49:28,688 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 59e56d11
>2012-12-17 08:49:28,708 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:28,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, SpmStopVDSCommand, log id: 59e56d11
>2012-12-17 08:49:28,904 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, ResetIrsVDSCommand, log id: 958fa36
>2012-12-17 08:49:28,915 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 449a9d13
>2012-12-17 08:49:28,915 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:28,915 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 449a9d13
>2012-12-17 08:49:29,023 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:49:29,054 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4662
>2012-12-17 08:49:29,055 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4662, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: aa10928
>2012-12-17 08:49:29,087 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] spmStart polling started: taskId = 33c5dd4d-7903-4f82-b753-91e4a7499ea8
>2012-12-17 08:49:30,109 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] spmStart polling ended: taskId = 33c5dd4d-7903-4f82-b753-91e4a7499ea8 task status = finished
>2012-12-17 08:49:30,127 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] spmStart polling ended, spm status: SPM
>2012-12-17 08:49:30,128 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=33c5dd4d-7903-4f82-b753-91e4a7499ea8), log id: b282da3
>2012-12-17 08:49:30,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, HSMClearTaskVDSCommand, log id: b282da3
>2012-12-17 08:49:30,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5683d847, log id: aa10928
>2012-12-17 08:49:30,190 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:49:30,240 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:30,242 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7008a527
>2012-12-17 08:49:30,261 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7008a527
>2012-12-17 08:49:30,261 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 583b9aa8
>2012-12-17 08:49:30,261 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [1f4324c3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:49:30,261 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4792dce7
>2012-12-17 08:49:30,261 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:30,262 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7bef4002
>2012-12-17 08:49:30,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7bef4002
>2012-12-17 08:49:30,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4792dce7
>2012-12-17 08:49:30,281 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [1f4324c3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:49:39,074 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [bb53630] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:49:39,180 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [2ba7b70d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:49:39,230 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 13ae6fbe
>2012-12-17 08:49:39,231 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 56324eb2
>2012-12-17 08:49:39,255 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:39,354 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:49:39,445 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] FINISH, SpmStopVDSCommand, log id: 56324eb2
>2012-12-17 08:49:39,477 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] FINISH, ResetIrsVDSCommand, log id: 13ae6fbe
>2012-12-17 08:49:39,487 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 41874892
>2012-12-17 08:49:39,488 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2ba7b70d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:39,488 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 41874892
>2012-12-17 08:49:49,559 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:49:49,598 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4663
>2012-12-17 08:49:49,599 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4663, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6c519af6
>2012-12-17 08:49:49,621 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] spmStart polling started: taskId = 3174d548-40a3-40b7-9020-a87b5479b4b9
>2012-12-17 08:49:50,645 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] spmStart polling ended: taskId = 3174d548-40a3-40b7-9020-a87b5479b4b9 task status = finished
>2012-12-17 08:49:50,663 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] spmStart polling ended, spm status: SPM
>2012-12-17 08:49:50,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3174d548-40a3-40b7-9020-a87b5479b4b9), log id: 2fafe119
>2012-12-17 08:49:50,682 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, HSMClearTaskVDSCommand, log id: 2fafe119
>2012-12-17 08:49:50,682 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@43e348c4, log id: 6c519af6
>2012-12-17 08:49:50,884 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:49:51,173 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [75b7a65b] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:49:51,173 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 736131e3
>2012-12-17 08:49:51,194 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:49:51,331 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-45) [63fee526] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:49:51,387 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 8677827
>2012-12-17 08:49:51,389 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 43e829f7
>2012-12-17 08:49:51,413 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [63fee526] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:51,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, SpmStopVDSCommand, log id: 43e829f7
>2012-12-17 08:49:51,594 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, ResetIrsVDSCommand, log id: 8677827
>2012-12-17 08:49:51,606 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6fe84348
>2012-12-17 08:49:51,606 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:51,606 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6fe84348
>2012-12-17 08:49:51,703 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:49:51,718 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4664
>2012-12-17 08:49:51,719 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4664, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 45b5af50
>2012-12-17 08:49:51,742 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling started: taskId = 47da8b06-04e0-4744-b6dc-1625b9fddd5f
>2012-12-17 08:49:52,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling ended: taskId = 47da8b06-04e0-4744-b6dc-1625b9fddd5f task status = finished
>2012-12-17 08:49:52,783 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling ended, spm status: SPM
>2012-12-17 08:49:52,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=47da8b06-04e0-4744-b6dc-1625b9fddd5f), log id: 1725fa2d
>2012-12-17 08:49:52,803 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, HSMClearTaskVDSCommand, log id: 1725fa2d
>2012-12-17 08:49:52,803 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e4bac3a, log id: 45b5af50
>2012-12-17 08:49:52,814 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:49:52,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:52,832 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1ddfa345
>2012-12-17 08:49:52,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ddfa345
>2012-12-17 08:49:52,851 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 736131e3
>2012-12-17 08:49:52,851 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [75b7a65b] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:49:52,852 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 45581655
>2012-12-17 08:49:52,852 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:49:52,853 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4fa22197
>2012-12-17 08:49:52,872 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4fa22197
>2012-12-17 08:49:52,872 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 45581655
>2012-12-17 08:49:52,872 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [75b7a65b] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:49:59,810 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:50:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [34599a45] Checking autorecoverable hosts
>2012-12-17 08:50:00,061 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [34599a45] Autorecovering 0 hosts
>2012-12-17 08:50:00,061 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [34599a45] Checking autorecoverable hosts done
>2012-12-17 08:50:00,061 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [34599a45] Autorecovering storage domains is disabled, skipping
>2012-12-17 08:50:02,246 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [6acb626c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:50:02,795 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-68) [1756fb88] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:50:02,870 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 290e135d
>2012-12-17 08:50:02,872 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 38a799b9
>2012-12-17 08:50:02,891 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:50:03,044 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, SpmStopVDSCommand, log id: 38a799b9
>2012-12-17 08:50:03,056 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, ResetIrsVDSCommand, log id: 290e135d
>2012-12-17 08:50:03,067 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 41c82391
>2012-12-17 08:50:03,067 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:50:03,067 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 41c82391
>2012-12-17 08:50:10,009 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1bff0c70] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:50:13,109 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [30867547] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:50:13,127 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [30867547] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4665
>2012-12-17 08:50:13,128 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [30867547] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4665, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4f0b70ca
>2012-12-17 08:50:13,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [30867547] spmStart polling started: taskId = 7f5cfcc3-b93f-451e-bd95-3af4d917abef
>2012-12-17 08:50:14,170 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [30867547] spmStart polling ended: taskId = 7f5cfcc3-b93f-451e-bd95-3af4d917abef task status = finished
>2012-12-17 08:50:14,188 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [30867547] spmStart polling ended, spm status: SPM
>2012-12-17 08:50:14,190 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [30867547] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7f5cfcc3-b93f-451e-bd95-3af4d917abef), log id: 419bf76
>2012-12-17 08:50:14,208 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [30867547] FINISH, HSMClearTaskVDSCommand, log id: 419bf76
>2012-12-17 08:50:14,208 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [30867547] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3668de14, log id: 4f0b70ca
>2012-12-17 08:50:14,219 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [30867547] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:50:14,269 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [28319bba] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:50:14,269 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 70a94de7
>2012-12-17 08:50:14,280 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [30867547] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:50:14,377 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-92) [69834199] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:50:14,471 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 59c3e568
>2012-12-17 08:50:14,472 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 467d720b
>2012-12-17 08:50:14,492 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [69834199] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:50:14,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, SpmStopVDSCommand, log id: 467d720b
>2012-12-17 08:50:14,674 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, ResetIrsVDSCommand, log id: 59c3e568
>2012-12-17 08:50:14,687 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1614edb
>2012-12-17 08:50:14,687 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:50:14,687 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1614edb
>2012-12-17 08:50:14,738 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:50:14,749 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4666
>2012-12-17 08:50:14,751 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4666, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 548f926b
>2012-12-17 08:50:14,773 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling started: taskId = 201dfa98-b024-412f-ae6a-8fee6a95dde1
>2012-12-17 08:50:15,796 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling ended: taskId = 201dfa98-b024-412f-ae6a-8fee6a95dde1 task status = finished
>2012-12-17 08:50:15,814 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling ended, spm status: SPM
>2012-12-17 08:50:15,815 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=201dfa98-b024-412f-ae6a-8fee6a95dde1), log id: 3dd06ab8
>2012-12-17 08:50:15,834 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, HSMClearTaskVDSCommand, log id: 3dd06ab8
>2012-12-17 08:50:15,834 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e0e75bc, log id: 548f926b
>2012-12-17 08:50:15,900 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:50:16,065 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:50:16,067 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 651cf3d2
>2012-12-17 08:50:16,098 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 651cf3d2
>2012-12-17 08:50:16,098 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 70a94de7
>2012-12-17 08:50:16,098 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [28319bba] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:50:16,098 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7c10555a
>2012-12-17 08:50:16,098 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:50:16,099 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 48b789cd
>2012-12-17 08:50:16,118 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 48b789cd
>2012-12-17 08:50:16,118 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7c10555a
>2012-12-17 08:50:16,118 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [28319bba] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:50:20,238 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:50:24,778 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [40a5fc59] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:50:24,935 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [5abe3acd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:50:25,034 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 670b31a5
>2012-12-17 08:50:25,036 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5573426a
>2012-12-17 08:50:25,056 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:50:25,663 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] FINISH, SpmStopVDSCommand, log id: 5573426a
>2012-12-17 08:50:26,388 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] FINISH, ResetIrsVDSCommand, log id: 670b31a5
>2012-12-17 08:50:26,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40a778e4
>2012-12-17 08:50:26,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5abe3acd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:50:26,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40a778e4
>2012-12-17 08:50:31,158 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:50:36,602 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [5b666f42] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:50:36,688 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [5b666f42] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4667
>2012-12-17 08:50:36,689 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4667, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 209d7ea0
>2012-12-17 08:50:36,711 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] spmStart polling started: taskId = e5da671c-aacf-4d8c-a8a7-7bc06a4d7e42
>2012-12-17 08:50:37,733 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] spmStart polling ended: taskId = e5da671c-aacf-4d8c-a8a7-7bc06a4d7e42 task status = finished
>2012-12-17 08:50:37,752 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] spmStart polling ended, spm status: SPM
>2012-12-17 08:50:37,753 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e5da671c-aacf-4d8c-a8a7-7bc06a4d7e42), log id: 790b80b0
>2012-12-17 08:50:37,772 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] FINISH, HSMClearTaskVDSCommand, log id: 790b80b0
>2012-12-17 08:50:37,772 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@622de0e5, log id: 209d7ea0
>2012-12-17 08:50:37,778 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [5b666f42] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:50:37,819 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) [e4c8c37] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:50:37,819 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7498f57b
>2012-12-17 08:50:37,835 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [5b666f42] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:50:37,965 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [3ef697d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:50:38,023 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 9902847
>2012-12-17 08:50:38,025 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 72adf961
>2012-12-17 08:50:38,046 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:50:38,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] FINISH, SpmStopVDSCommand, log id: 72adf961
>2012-12-17 08:50:38,211 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] FINISH, ResetIrsVDSCommand, log id: 9902847
>2012-12-17 08:50:38,222 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c2fb07f
>2012-12-17 08:50:38,222 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3ef697d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:50:38,222 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c2fb07f
>2012-12-17 08:50:38,277 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:50:38,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4668
>2012-12-17 08:50:38,288 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4668, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7722fc7
>2012-12-17 08:50:38,310 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] spmStart polling started: taskId = 41b3e09c-c53f-448d-baaf-d51e676b3ec1
>2012-12-17 08:50:39,331 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] spmStart polling ended: taskId = 41b3e09c-c53f-448d-baaf-d51e676b3ec1 task status = finished
>2012-12-17 08:50:39,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] spmStart polling ended, spm status: SPM
>2012-12-17 08:50:39,351 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=41b3e09c-c53f-448d-baaf-d51e676b3ec1), log id: 559511e2
>2012-12-17 08:50:39,369 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, HSMClearTaskVDSCommand, log id: 559511e2
>2012-12-17 08:50:39,369 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2dc93853, log id: 7722fc7
>2012-12-17 08:50:39,379 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:50:39,425 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:50:39,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5f2902b6
>2012-12-17 08:50:39,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5f2902b6
>2012-12-17 08:50:39,449 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7498f57b
>2012-12-17 08:50:39,449 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) [e4c8c37] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:50:39,449 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5b2e281f
>2012-12-17 08:50:39,449 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:50:39,451 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 72eab901
>2012-12-17 08:50:39,469 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 72eab901
>2012-12-17 08:50:39,469 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5b2e281f
>2012-12-17 08:50:39,469 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) [e4c8c37] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:50:41,345 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [6c95e244] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:50:48,341 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [4c63d330] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:50:48,443 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-34) [5dcd105e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:50:48,506 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3fef855f
>2012-12-17 08:50:48,507 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7cece9e6
>2012-12-17 08:50:48,527 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:50:48,794 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, SpmStopVDSCommand, log id: 7cece9e6
>2012-12-17 08:50:48,806 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, ResetIrsVDSCommand, log id: 3fef855f
>2012-12-17 08:50:48,816 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bafbf3d
>2012-12-17 08:50:48,816 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [5dcd105e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:50:48,817 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bafbf3d
>2012-12-17 08:50:51,875 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:50:58,891 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [43f46040] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:50:58,908 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [43f46040] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4669
>2012-12-17 08:50:58,910 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4669, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 467b21e3
>2012-12-17 08:50:58,932 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] spmStart polling started: taskId = ca01353f-a28b-4611-82be-a3becdf565f2
>2012-12-17 08:50:59,954 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] spmStart polling ended: taskId = ca01353f-a28b-4611-82be-a3becdf565f2 task status = finished
>2012-12-17 08:50:59,972 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] spmStart polling ended, spm status: SPM
>2012-12-17 08:50:59,973 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ca01353f-a28b-4611-82be-a3becdf565f2), log id: 1b040dbd
>2012-12-17 08:51:00,005 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, HSMClearTaskVDSCommand, log id: 1b040dbd
>2012-12-17 08:51:00,005 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1ad60580, log id: 467b21e3
>2012-12-17 08:51:00,011 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [43f46040] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:51:00,046 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [3e0f12eb] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:51:00,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1a5488f4
>2012-12-17 08:51:00,073 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [43f46040] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:51:01,034 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [dd34b5e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:51:01,225 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [dd34b5e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 598aee86
>2012-12-17 08:51:01,227 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [dd34b5e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1667a46d
>2012-12-17 08:51:01,248 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [dd34b5e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:01,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [dd34b5e] FINISH, SpmStopVDSCommand, log id: 1667a46d
>2012-12-17 08:51:01,394 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [dd34b5e] FINISH, ResetIrsVDSCommand, log id: 598aee86
>2012-12-17 08:51:01,405 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [dd34b5e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 421a6141
>2012-12-17 08:51:01,405 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [dd34b5e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:01,405 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [dd34b5e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 421a6141
>2012-12-17 08:51:01,472 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:51:01,486 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4670
>2012-12-17 08:51:01,487 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4670, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6890411b
>2012-12-17 08:51:01,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] spmStart polling started: taskId = f7fb0fba-a7f1-4bf8-a4a2-dbae638279ea
>2012-12-17 08:51:02,530 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] spmStart polling ended: taskId = f7fb0fba-a7f1-4bf8-a4a2-dbae638279ea task status = finished
>2012-12-17 08:51:02,549 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] spmStart polling ended, spm status: SPM
>2012-12-17 08:51:02,550 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f7fb0fba-a7f1-4bf8-a4a2-dbae638279ea), log id: 6dc764c4
>2012-12-17 08:51:02,569 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, HSMClearTaskVDSCommand, log id: 6dc764c4
>2012-12-17 08:51:02,569 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3da8b82a, log id: 6890411b
>2012-12-17 08:51:02,595 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:51:02,644 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:02,647 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: d4ae3ec
>2012-12-17 08:51:02,667 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: d4ae3ec
>2012-12-17 08:51:02,667 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1a5488f4
>2012-12-17 08:51:02,667 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [3e0f12eb] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:51:02,667 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2fb9a1c5
>2012-12-17 08:51:02,667 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:02,668 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3dd94d91
>2012-12-17 08:51:02,687 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3dd94d91
>2012-12-17 08:51:02,687 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2fb9a1c5
>2012-12-17 08:51:02,687 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [3e0f12eb] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:51:11,548 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [726a9e10] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:51:11,693 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [45590d60] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:51:11,739 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [45590d60] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7312cb8b
>2012-12-17 08:51:11,741 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [45590d60] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7a77a8af
>2012-12-17 08:51:11,761 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [45590d60] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:11,916 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [45590d60] FINISH, SpmStopVDSCommand, log id: 7a77a8af
>2012-12-17 08:51:11,929 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [45590d60] FINISH, ResetIrsVDSCommand, log id: 7312cb8b
>2012-12-17 08:51:11,942 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [45590d60] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7859e73e
>2012-12-17 08:51:11,942 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [45590d60] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:11,942 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [45590d60] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7859e73e
>2012-12-17 08:51:12,338 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4beddd99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:51:22,083 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3eec3b83] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:51:22,113 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3eec3b83] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4671
>2012-12-17 08:51:22,114 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4671, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 298af872
>2012-12-17 08:51:22,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] spmStart polling started: taskId = cae355e4-a638-4ff7-add6-41a6c91bea68
>2012-12-17 08:51:23,171 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] spmStart polling ended: taskId = cae355e4-a638-4ff7-add6-41a6c91bea68 task status = finished
>2012-12-17 08:51:23,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] spmStart polling ended, spm status: SPM
>2012-12-17 08:51:23,190 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cae355e4-a638-4ff7-add6-41a6c91bea68), log id: 53d4beca
>2012-12-17 08:51:23,209 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] FINISH, HSMClearTaskVDSCommand, log id: 53d4beca
>2012-12-17 08:51:23,209 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3439eac5, log id: 298af872
>2012-12-17 08:51:23,248 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3eec3b83] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:51:23,318 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:51:23,318 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6cb88bf1
>2012-12-17 08:51:23,331 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3eec3b83] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:51:23,467 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [7c1d0790] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:51:23,609 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [7c1d0790] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2b94a9eb
>2012-12-17 08:51:23,610 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [7c1d0790] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5de3b838
>2012-12-17 08:51:23,630 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [7c1d0790] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:24,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [7c1d0790] FINISH, SpmStopVDSCommand, log id: 5de3b838
>2012-12-17 08:51:24,402 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [7c1d0790] FINISH, ResetIrsVDSCommand, log id: 2b94a9eb
>2012-12-17 08:51:24,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [7c1d0790] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 54fc8a0b
>2012-12-17 08:51:24,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [7c1d0790] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:24,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [7c1d0790] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 54fc8a0b
>2012-12-17 08:51:24,547 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:51:24,580 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4672
>2012-12-17 08:51:24,593 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4672, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 52c928a6
>2012-12-17 08:51:24,615 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling started: taskId = 8ee30c57-2f44-4a0a-953e-1896ad25efb7
>2012-12-17 08:51:25,637 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling ended: taskId = 8ee30c57-2f44-4a0a-953e-1896ad25efb7 task status = finished
>2012-12-17 08:51:25,655 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling ended, spm status: SPM
>2012-12-17 08:51:25,657 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8ee30c57-2f44-4a0a-953e-1896ad25efb7), log id: 10e93c40
>2012-12-17 08:51:25,675 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMClearTaskVDSCommand, log id: 10e93c40
>2012-12-17 08:51:25,675 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4072966c, log id: 52c928a6
>2012-12-17 08:51:25,977 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:51:26,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:26,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1393007d
>2012-12-17 08:51:26,179 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1393007d
>2012-12-17 08:51:26,179 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6cb88bf1
>2012-12-17 08:51:26,180 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:51:26,180 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 18051895
>2012-12-17 08:51:26,180 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:26,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7faa7d13
>2012-12-17 08:51:26,199 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7faa7d13
>2012-12-17 08:51:26,199 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 18051895
>2012-12-17 08:51:26,199 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:51:32,836 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [107aac36] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:51:34,582 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [2de700da] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:51:34,792 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [6d961413] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:51:34,882 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3eb436e3
>2012-12-17 08:51:34,884 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1e94f6f9
>2012-12-17 08:51:34,904 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [6d961413] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:35,055 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, SpmStopVDSCommand, log id: 1e94f6f9
>2012-12-17 08:51:35,080 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, ResetIrsVDSCommand, log id: 3eb436e3
>2012-12-17 08:51:35,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6b17ab52
>2012-12-17 08:51:35,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [6d961413] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:35,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6b17ab52
>2012-12-17 08:51:43,015 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:51:45,163 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [369f35cc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:51:45,185 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [369f35cc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4673
>2012-12-17 08:51:45,186 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4673, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 72ddc50
>2012-12-17 08:51:45,209 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] spmStart polling started: taskId = aeded897-fd9b-46e7-9fbf-49ee29f65840
>2012-12-17 08:51:46,231 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] spmStart polling ended: taskId = aeded897-fd9b-46e7-9fbf-49ee29f65840 task status = finished
>2012-12-17 08:51:46,249 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] spmStart polling ended, spm status: SPM
>2012-12-17 08:51:46,250 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=aeded897-fd9b-46e7-9fbf-49ee29f65840), log id: 6c72c6ff
>2012-12-17 08:51:46,268 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] FINISH, HSMClearTaskVDSCommand, log id: 6c72c6ff
>2012-12-17 08:51:46,268 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6366c6f8, log id: 72ddc50
>2012-12-17 08:51:46,279 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [369f35cc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:51:46,312 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1bff0c70] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:51:46,312 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 68588ca1
>2012-12-17 08:51:46,447 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [369f35cc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:51:46,584 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [32c2c5a9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:51:46,693 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1219563a
>2012-12-17 08:51:46,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 188b444c
>2012-12-17 08:51:46,715 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:47,135 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] FINISH, SpmStopVDSCommand, log id: 188b444c
>2012-12-17 08:51:47,159 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] FINISH, ResetIrsVDSCommand, log id: 1219563a
>2012-12-17 08:51:47,170 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2983705
>2012-12-17 08:51:47,170 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [32c2c5a9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:47,170 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2983705
>2012-12-17 08:51:47,218 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1bff0c70] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:51:47,258 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1bff0c70] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4674
>2012-12-17 08:51:47,260 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4674, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2a58acfc
>2012-12-17 08:51:47,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] spmStart polling started: taskId = 4a1f375f-a968-4edd-9975-a7fae867e08a
>2012-12-17 08:51:48,303 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] spmStart polling ended: taskId = 4a1f375f-a968-4edd-9975-a7fae867e08a task status = finished
>2012-12-17 08:51:48,321 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] spmStart polling ended, spm status: SPM
>2012-12-17 08:51:48,322 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4a1f375f-a968-4edd-9975-a7fae867e08a), log id: 68291869
>2012-12-17 08:51:48,341 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, HSMClearTaskVDSCommand, log id: 68291869
>2012-12-17 08:51:48,341 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6201caeb, log id: 2a58acfc
>2012-12-17 08:51:48,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1bff0c70] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:51:48,381 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:48,383 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6f776570
>2012-12-17 08:51:48,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f776570
>2012-12-17 08:51:48,405 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 68588ca1
>2012-12-17 08:51:48,405 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1bff0c70] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:51:48,405 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 56ab5a0f
>2012-12-17 08:51:48,406 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:48,407 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 73e34196
>2012-12-17 08:51:48,426 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 73e34196
>2012-12-17 08:51:48,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 56ab5a0f
>2012-12-17 08:51:48,426 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1bff0c70] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:51:53,254 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:51:57,263 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [2b22186] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:51:57,409 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-36) [4793062d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:51:57,558 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [4793062d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 720fb799
>2012-12-17 08:51:57,559 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [4793062d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 212f4563
>2012-12-17 08:51:57,579 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [4793062d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:57,768 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [4793062d] FINISH, SpmStopVDSCommand, log id: 212f4563
>2012-12-17 08:51:57,781 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [4793062d] FINISH, ResetIrsVDSCommand, log id: 720fb799
>2012-12-17 08:51:57,791 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [4793062d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6dfdefc6
>2012-12-17 08:51:57,792 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [4793062d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:51:57,792 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [4793062d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6dfdefc6
>2012-12-17 08:52:03,473 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [d00852e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:52:07,875 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:52:07,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4675
>2012-12-17 08:52:07,888 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4675, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: edd6759
>2012-12-17 08:52:07,910 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] spmStart polling started: taskId = 14cd4bd5-bd6a-4e22-81f3-da19916dcbd7
>2012-12-17 08:52:08,932 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] spmStart polling ended: taskId = 14cd4bd5-bd6a-4e22-81f3-da19916dcbd7 task status = finished
>2012-12-17 08:52:08,951 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] spmStart polling ended, spm status: SPM
>2012-12-17 08:52:08,952 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=14cd4bd5-bd6a-4e22-81f3-da19916dcbd7), log id: 461e5ede
>2012-12-17 08:52:08,970 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, HSMClearTaskVDSCommand, log id: 461e5ede
>2012-12-17 08:52:08,970 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@73e664c8, log id: edd6759
>2012-12-17 08:52:08,981 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:52:09,021 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [15ef2937] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:52:09,021 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4a08efa5
>2012-12-17 08:52:09,051 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:52:09,178 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [5e38fcf2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:52:09,238 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 362177e3
>2012-12-17 08:52:09,239 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 79692fa8
>2012-12-17 08:52:09,260 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:09,394 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, SpmStopVDSCommand, log id: 79692fa8
>2012-12-17 08:52:09,416 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, ResetIrsVDSCommand, log id: 362177e3
>2012-12-17 08:52:09,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a047226
>2012-12-17 08:52:09,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:09,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a047226
>2012-12-17 08:52:09,478 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [15ef2937] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:52:09,503 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [15ef2937] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4676
>2012-12-17 08:52:09,504 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4676, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 347f8d5e
>2012-12-17 08:52:09,525 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] spmStart polling started: taskId = 59f89343-8723-4562-904c-f5ad0dedd1be
>2012-12-17 08:52:10,547 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] spmStart polling ended: taskId = 59f89343-8723-4562-904c-f5ad0dedd1be task status = finished
>2012-12-17 08:52:10,566 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] spmStart polling ended, spm status: SPM
>2012-12-17 08:52:10,567 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=59f89343-8723-4562-904c-f5ad0dedd1be), log id: 14cf435
>2012-12-17 08:52:10,585 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, HSMClearTaskVDSCommand, log id: 14cf435
>2012-12-17 08:52:10,585 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6334c43d, log id: 347f8d5e
>2012-12-17 08:52:11,879 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [15ef2937] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:52:11,977 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:11,979 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 641db1bc
>2012-12-17 08:52:11,998 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 641db1bc
>2012-12-17 08:52:11,998 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4a08efa5
>2012-12-17 08:52:11,998 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [15ef2937] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:52:11,998 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 597fe0fd
>2012-12-17 08:52:11,998 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:12,000 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5705fbd7
>2012-12-17 08:52:12,018 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5705fbd7
>2012-12-17 08:52:12,018 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 597fe0fd
>2012-12-17 08:52:12,018 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [15ef2937] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:52:13,750 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:52:19,516 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [37434c11] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:52:19,674 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [61136831] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:52:19,743 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 18969ea4
>2012-12-17 08:52:19,744 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 25fddbea
>2012-12-17 08:52:19,764 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [61136831] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:19,935 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, SpmStopVDSCommand, log id: 25fddbea
>2012-12-17 08:52:19,972 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, ResetIrsVDSCommand, log id: 18969ea4
>2012-12-17 08:52:19,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1ee7b32c
>2012-12-17 08:52:19,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:19,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1ee7b32c
>2012-12-17 08:52:23,950 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:52:30,083 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:52:30,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4677
>2012-12-17 08:52:30,097 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4677, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 10149c93
>2012-12-17 08:52:30,118 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] spmStart polling started: taskId = 9cf9966d-4f49-467f-b941-49e6cba3d5bb
>2012-12-17 08:52:31,140 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] spmStart polling ended: taskId = 9cf9966d-4f49-467f-b941-49e6cba3d5bb task status = finished
>2012-12-17 08:52:31,159 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] spmStart polling ended, spm status: SPM
>2012-12-17 08:52:31,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9cf9966d-4f49-467f-b941-49e6cba3d5bb), log id: 79d3af9d
>2012-12-17 08:52:31,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, HSMClearTaskVDSCommand, log id: 79d3af9d
>2012-12-17 08:52:31,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@38aeaae7, log id: 10149c93
>2012-12-17 08:52:31,204 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:52:31,569 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [155e23b1] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:52:31,569 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 31b0b7a0
>2012-12-17 08:52:31,661 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:52:31,924 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:52:32,093 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7bccd9f9
>2012-12-17 08:52:32,095 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1c99c56b
>2012-12-17 08:52:32,114 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:32,252 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SpmStopVDSCommand, log id: 1c99c56b
>2012-12-17 08:52:32,313 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, ResetIrsVDSCommand, log id: 7bccd9f9
>2012-12-17 08:52:32,324 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ff1f3be
>2012-12-17 08:52:32,324 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:32,324 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ff1f3be
>2012-12-17 08:52:32,410 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:52:32,420 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4678
>2012-12-17 08:52:32,421 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4678, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3a132d9
>2012-12-17 08:52:32,490 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] spmStart polling started: taskId = 6950ea93-afd3-49c9-bff1-833f6ee2b905
>2012-12-17 08:52:33,512 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] spmStart polling ended: taskId = 6950ea93-afd3-49c9-bff1-833f6ee2b905 task status = finished
>2012-12-17 08:52:33,530 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] spmStart polling ended, spm status: SPM
>2012-12-17 08:52:33,532 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6950ea93-afd3-49c9-bff1-833f6ee2b905), log id: be3048a
>2012-12-17 08:52:33,550 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, HSMClearTaskVDSCommand, log id: be3048a
>2012-12-17 08:52:33,551 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@653f061e, log id: 3a132d9
>2012-12-17 08:52:33,563 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:52:33,578 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:33,582 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 36f7bdd1
>2012-12-17 08:52:33,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 36f7bdd1
>2012-12-17 08:52:33,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 31b0b7a0
>2012-12-17 08:52:33,601 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [155e23b1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:52:33,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 242c02fc
>2012-12-17 08:52:33,602 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:33,603 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 66c07d6a
>2012-12-17 08:52:33,621 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 66c07d6a
>2012-12-17 08:52:33,621 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 242c02fc
>2012-12-17 08:52:33,621 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [155e23b1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:52:34,158 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [32c2c5a9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:52:42,467 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:52:42,567 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [164629b0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:52:42,620 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [164629b0] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 515e0f1e
>2012-12-17 08:52:42,621 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [164629b0] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5ee04a3a
>2012-12-17 08:52:42,641 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [164629b0] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:43,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [164629b0] FINISH, SpmStopVDSCommand, log id: 5ee04a3a
>2012-12-17 08:52:43,183 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [164629b0] FINISH, ResetIrsVDSCommand, log id: 515e0f1e
>2012-12-17 08:52:43,195 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [164629b0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 48b972e6
>2012-12-17 08:52:43,195 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [164629b0] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:43,195 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [164629b0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 48b972e6
>2012-12-17 08:52:44,319 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:52:53,263 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [369a76fa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:52:53,340 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [369a76fa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4679
>2012-12-17 08:52:53,341 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4679, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 31fde7a6
>2012-12-17 08:52:53,365 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] spmStart polling started: taskId = 4cfd75cb-5dfb-46f9-bdce-e657fbf19c8a
>2012-12-17 08:52:54,387 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] spmStart polling ended: taskId = 4cfd75cb-5dfb-46f9-bdce-e657fbf19c8a task status = finished
>2012-12-17 08:52:54,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] spmStart polling ended, spm status: SPM
>2012-12-17 08:52:54,406 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4cfd75cb-5dfb-46f9-bdce-e657fbf19c8a), log id: 7fc9c942
>2012-12-17 08:52:54,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] FINISH, HSMClearTaskVDSCommand, log id: 7fc9c942
>2012-12-17 08:52:54,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@18bc4d99, log id: 31fde7a6
>2012-12-17 08:52:54,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [369a76fa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:52:54,511 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:52:54,511 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1c7e99ef
>2012-12-17 08:52:54,553 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [369a76fa] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:52:54,768 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [214f1ee4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:52:54,826 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4b77eddd
>2012-12-17 08:52:54,827 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 65f44d4d
>2012-12-17 08:52:54,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:54,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, SpmStopVDSCommand, log id: 65f44d4d
>2012-12-17 08:52:55,052 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, ResetIrsVDSCommand, log id: 4b77eddd
>2012-12-17 08:52:55,064 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 542352da
>2012-12-17 08:52:55,064 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:55,064 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 542352da
>2012-12-17 08:52:55,112 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:52:55,141 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4680
>2012-12-17 08:52:55,142 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4680, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b932c37
>2012-12-17 08:52:55,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] spmStart polling started: taskId = 8bf086ef-dddf-4b28-b4c8-b9fea5ee01f1
>2012-12-17 08:52:56,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] spmStart polling ended: taskId = 8bf086ef-dddf-4b28-b4c8-b9fea5ee01f1 task status = finished
>2012-12-17 08:52:56,208 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] spmStart polling ended, spm status: SPM
>2012-12-17 08:52:56,210 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8bf086ef-dddf-4b28-b4c8-b9fea5ee01f1), log id: 106556c
>2012-12-17 08:52:56,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMClearTaskVDSCommand, log id: 106556c
>2012-12-17 08:52:56,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@46d225fa, log id: 6b932c37
>2012-12-17 08:52:56,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:52:56,445 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:56,447 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 548138fd
>2012-12-17 08:52:56,465 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 548138fd
>2012-12-17 08:52:56,466 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1c7e99ef
>2012-12-17 08:52:56,466 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:52:56,466 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6fc49562
>2012-12-17 08:52:56,466 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:52:56,467 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 98f257e
>2012-12-17 08:52:56,485 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 98f257e
>2012-12-17 08:52:56,485 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6fc49562
>2012-12-17 08:52:56,486 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:53:04,766 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:53:05,147 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [1a40d676] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:53:05,853 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [33e4b3f5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:53:06,749 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2d3bfd78
>2012-12-17 08:53:06,751 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 76dc153
>2012-12-17 08:53:06,771 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:53:06,982 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] FINISH, SpmStopVDSCommand, log id: 76dc153
>2012-12-17 08:53:06,993 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] FINISH, ResetIrsVDSCommand, log id: 2d3bfd78
>2012-12-17 08:53:07,004 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 591d4c18
>2012-12-17 08:53:07,004 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [33e4b3f5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:53:07,004 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 591d4c18
>2012-12-17 08:53:14,965 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:53:17,053 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [32ec1ec3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:53:17,068 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [32ec1ec3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4681
>2012-12-17 08:53:17,069 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4681, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 233d4423
>2012-12-17 08:53:17,090 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] spmStart polling started: taskId = 53859cae-6fd0-4895-ae39-c20cc465bd9c
>2012-12-17 08:53:19,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] spmStart polling ended: taskId = 53859cae-6fd0-4895-ae39-c20cc465bd9c task status = finished
>2012-12-17 08:53:19,154 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] spmStart polling ended, spm status: SPM
>2012-12-17 08:53:19,156 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=53859cae-6fd0-4895-ae39-c20cc465bd9c), log id: 195dff7f
>2012-12-17 08:53:19,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] FINISH, HSMClearTaskVDSCommand, log id: 195dff7f
>2012-12-17 08:53:19,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a30f7db, log id: 233d4423
>2012-12-17 08:53:19,193 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [32ec1ec3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:53:19,298 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [39a5c05e] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:53:19,299 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3d325152
>2012-12-17 08:53:19,332 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [32ec1ec3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:53:19,478 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [490300dd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:53:19,554 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [490300dd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3a4aac1
>2012-12-17 08:53:19,555 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [490300dd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2f5ff814
>2012-12-17 08:53:19,575 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [490300dd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:53:19,747 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [490300dd] FINISH, SpmStopVDSCommand, log id: 2f5ff814
>2012-12-17 08:53:19,763 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [490300dd] FINISH, ResetIrsVDSCommand, log id: 3a4aac1
>2012-12-17 08:53:19,774 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [490300dd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6545a35
>2012-12-17 08:53:19,774 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [490300dd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:53:19,774 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [490300dd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6545a35
>2012-12-17 08:53:19,835 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:53:19,846 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4682
>2012-12-17 08:53:19,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4682, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2cf1dbac
>2012-12-17 08:53:19,868 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling started: taskId = a8576c80-e37c-4742-8337-6e1f85043f30
>2012-12-17 08:53:20,890 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling ended: taskId = a8576c80-e37c-4742-8337-6e1f85043f30 task status = finished
>2012-12-17 08:53:20,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling ended, spm status: SPM
>2012-12-17 08:53:20,910 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a8576c80-e37c-4742-8337-6e1f85043f30), log id: ba01558
>2012-12-17 08:53:20,928 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, HSMClearTaskVDSCommand, log id: ba01558
>2012-12-17 08:53:20,928 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3ad94a5e, log id: 2cf1dbac
>2012-12-17 08:53:21,384 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:53:21,537 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:53:21,540 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2693cda3
>2012-12-17 08:53:21,560 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2693cda3
>2012-12-17 08:53:21,560 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3d325152
>2012-12-17 08:53:21,560 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [39a5c05e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:53:21,560 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 31a19c1e
>2012-12-17 08:53:21,560 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:53:21,561 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 60c585fe
>2012-12-17 08:53:21,579 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 60c585fe
>2012-12-17 08:53:21,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 31a19c1e
>2012-12-17 08:53:21,579 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [39a5c05e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:53:25,223 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:53:29,885 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [780c369f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:53:30,068 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [5bd08b5f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:53:30,172 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [5bd08b5f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 39c5503
>2012-12-17 08:53:30,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [5bd08b5f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 45635c70
>2012-12-17 08:53:30,194 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [5bd08b5f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:53:31,131 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [5bd08b5f] FINISH, SpmStopVDSCommand, log id: 45635c70
>2012-12-17 08:53:31,238 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [5bd08b5f] FINISH, ResetIrsVDSCommand, log id: 39c5503
>2012-12-17 08:53:31,249 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [5bd08b5f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4069f938
>2012-12-17 08:53:31,250 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5bd08b5f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:53:31,250 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [5bd08b5f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4069f938
>2012-12-17 08:53:36,328 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:53:41,739 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [337c1daa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:53:41,762 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [337c1daa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4683
>2012-12-17 08:53:41,763 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4683, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 89419f
>2012-12-17 08:53:41,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] spmStart polling started: taskId = 30d8b6fc-083c-451e-a99c-2f08b2142a03
>2012-12-17 08:53:42,806 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] spmStart polling ended: taskId = 30d8b6fc-083c-451e-a99c-2f08b2142a03 task status = finished
>2012-12-17 08:53:42,824 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] spmStart polling ended, spm status: SPM
>2012-12-17 08:53:42,826 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=30d8b6fc-083c-451e-a99c-2f08b2142a03), log id: a545803
>2012-12-17 08:53:42,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] FINISH, HSMClearTaskVDSCommand, log id: a545803
>2012-12-17 08:53:42,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@275e3241, log id: 89419f
>2012-12-17 08:53:42,878 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [337c1daa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:53:42,974 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [214f1ee4] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:53:42,974 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7dc13231
>2012-12-17 08:53:42,992 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [337c1daa] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:53:43,136 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-18) [5a014659] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:53:43,186 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [5a014659] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4a981fcb
>2012-12-17 08:53:43,188 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [5a014659] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5502fa2d
>2012-12-17 08:53:43,211 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [5a014659] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:53:43,804 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [5a014659] FINISH, SpmStopVDSCommand, log id: 5502fa2d
>2012-12-17 08:53:43,817 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [5a014659] FINISH, ResetIrsVDSCommand, log id: 4a981fcb
>2012-12-17 08:53:43,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [5a014659] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6beef539
>2012-12-17 08:53:43,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [5a014659] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:53:43,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [5a014659] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6beef539
>2012-12-17 08:53:43,874 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:53:43,912 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4684
>2012-12-17 08:53:43,913 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4684, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7090bab6
>2012-12-17 08:53:43,936 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] spmStart polling started: taskId = ebfc0b96-3694-42bf-a60c-f4c0043a02d5
>2012-12-17 08:53:44,957 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] spmStart polling ended: taskId = ebfc0b96-3694-42bf-a60c-f4c0043a02d5 task status = finished
>2012-12-17 08:53:44,975 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] spmStart polling ended, spm status: SPM
>2012-12-17 08:53:44,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ebfc0b96-3694-42bf-a60c-f4c0043a02d5), log id: 731baae7
>2012-12-17 08:53:44,994 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, HSMClearTaskVDSCommand, log id: 731baae7
>2012-12-17 08:53:44,995 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1d715ebb, log id: 7090bab6
>2012-12-17 08:53:45,018 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:53:45,038 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:53:45,040 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 79176827
>2012-12-17 08:53:45,058 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 79176827
>2012-12-17 08:53:45,059 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7dc13231
>2012-12-17 08:53:45,059 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [214f1ee4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:53:45,059 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6fb7168c
>2012-12-17 08:53:45,059 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:53:45,060 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1ffaddd6
>2012-12-17 08:53:45,078 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ffaddd6
>2012-12-17 08:53:45,078 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6fb7168c
>2012-12-17 08:53:45,078 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [214f1ee4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:53:46,948 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:53:53,930 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [6cc0b445] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:53:54,086 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [37630ba1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:53:54,147 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [37630ba1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 36970edb
>2012-12-17 08:53:54,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [37630ba1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 21c010bb
>2012-12-17 08:53:54,168 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [37630ba1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:53:54,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [37630ba1] FINISH, SpmStopVDSCommand, log id: 21c010bb
>2012-12-17 08:53:54,876 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [37630ba1] FINISH, ResetIrsVDSCommand, log id: 36970edb
>2012-12-17 08:53:54,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [37630ba1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 472ccf44
>2012-12-17 08:53:54,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [37630ba1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:53:54,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [37630ba1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 472ccf44
>2012-12-17 08:53:57,156 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [107aac36] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:54:04,936 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [71e1da67] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:54:04,950 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [71e1da67] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4685
>2012-12-17 08:54:04,951 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4685, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6dd47c44
>2012-12-17 08:54:04,973 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] spmStart polling started: taskId = f186d2ae-d377-444c-8a41-d4e92ff735d9
>2012-12-17 08:54:05,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] spmStart polling ended: taskId = f186d2ae-d377-444c-8a41-d4e92ff735d9 task status = finished
>2012-12-17 08:54:06,014 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] spmStart polling ended, spm status: SPM
>2012-12-17 08:54:06,015 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f186d2ae-d377-444c-8a41-d4e92ff735d9), log id: 751210b9
>2012-12-17 08:54:06,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] FINISH, HSMClearTaskVDSCommand, log id: 751210b9
>2012-12-17 08:54:06,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@75081dba, log id: 6dd47c44
>2012-12-17 08:54:06,305 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [71e1da67] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:54:06,673 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [3e3e3ebc] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:54:06,673 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 109df0bf
>2012-12-17 08:54:06,730 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [71e1da67] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:54:06,888 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:54:06,935 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4da03343
>2012-12-17 08:54:06,937 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 29604dbf
>2012-12-17 08:54:06,958 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:07,089 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, SpmStopVDSCommand, log id: 29604dbf
>2012-12-17 08:54:07,130 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, ResetIrsVDSCommand, log id: 4da03343
>2012-12-17 08:54:07,140 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: acaa2c6
>2012-12-17 08:54:07,141 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:07,141 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: acaa2c6
>2012-12-17 08:54:07,211 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:54:07,223 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4686
>2012-12-17 08:54:07,225 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4686, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6d4cb4aa
>2012-12-17 08:54:07,248 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] spmStart polling started: taskId = 02d9e6de-f0e5-4992-a5df-5b33b617e1bd
>2012-12-17 08:54:08,270 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] spmStart polling ended: taskId = 02d9e6de-f0e5-4992-a5df-5b33b617e1bd task status = finished
>2012-12-17 08:54:08,288 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] spmStart polling ended, spm status: SPM
>2012-12-17 08:54:08,289 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=02d9e6de-f0e5-4992-a5df-5b33b617e1bd), log id: 6b9ed15a
>2012-12-17 08:54:08,307 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, HSMClearTaskVDSCommand, log id: 6b9ed15a
>2012-12-17 08:54:08,307 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7c2a2e9, log id: 6d4cb4aa
>2012-12-17 08:54:08,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:54:08,348 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:08,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 496c5c1e
>2012-12-17 08:54:08,372 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 496c5c1e
>2012-12-17 08:54:08,372 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 109df0bf
>2012-12-17 08:54:08,372 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [3e3e3ebc] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:54:08,372 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 169df2bb
>2012-12-17 08:54:08,372 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:08,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 556ea340
>2012-12-17 08:54:08,391 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 556ea340
>2012-12-17 08:54:08,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 169df2bb
>2012-12-17 08:54:08,391 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [3e3e3ebc] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:54:17,250 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4232612d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:54:17,432 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-75) [79e50d62] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:54:17,486 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [79e50d62] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6a4335
>2012-12-17 08:54:17,487 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [79e50d62] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2f94e7cc
>2012-12-17 08:54:17,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [79e50d62] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:17,598 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:54:18,194 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [79e50d62] FINISH, SpmStopVDSCommand, log id: 2f94e7cc
>2012-12-17 08:54:18,221 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [79e50d62] FINISH, ResetIrsVDSCommand, log id: 6a4335
>2012-12-17 08:54:18,231 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [79e50d62] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 31ea2f20
>2012-12-17 08:54:18,231 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [79e50d62] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:18,231 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [79e50d62] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 31ea2f20
>2012-12-17 08:54:28,288 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [d00852e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:54:28,297 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [d00852e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4687
>2012-12-17 08:54:28,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [d00852e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4687, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 37e53615
>2012-12-17 08:54:28,321 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [d00852e] spmStart polling started: taskId = 6cb8e301-8614-4f2e-95c5-a31588ff371f
>2012-12-17 08:54:29,352 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [d00852e] spmStart polling ended: taskId = 6cb8e301-8614-4f2e-95c5-a31588ff371f task status = finished
>2012-12-17 08:54:29,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [d00852e] spmStart polling ended, spm status: SPM
>2012-12-17 08:54:29,376 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [d00852e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6cb8e301-8614-4f2e-95c5-a31588ff371f), log id: 5d608511
>2012-12-17 08:54:29,396 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [d00852e] FINISH, HSMClearTaskVDSCommand, log id: 5d608511
>2012-12-17 08:54:29,396 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [d00852e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@33908c84, log id: 37e53615
>2012-12-17 08:54:29,420 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [d00852e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:54:29,484 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:54:29,485 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7e595dac
>2012-12-17 08:54:29,501 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [d00852e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:54:29,706 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [15510694] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:54:29,788 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [15510694] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2888804e
>2012-12-17 08:54:29,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [15510694] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7a9ab0ae
>2012-12-17 08:54:29,827 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [15510694] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:30,012 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [15510694] FINISH, SpmStopVDSCommand, log id: 7a9ab0ae
>2012-12-17 08:54:30,034 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [15510694] FINISH, ResetIrsVDSCommand, log id: 2888804e
>2012-12-17 08:54:30,045 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [15510694] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4514948e
>2012-12-17 08:54:30,045 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [15510694] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:30,045 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [15510694] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4514948e
>2012-12-17 08:54:30,104 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:54:30,118 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4688
>2012-12-17 08:54:30,120 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4688, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5692ab98
>2012-12-17 08:54:30,142 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling started: taskId = e2ef5666-365a-4fac-8a08-89fcfd4b0a70
>2012-12-17 08:54:31,164 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended: taskId = e2ef5666-365a-4fac-8a08-89fcfd4b0a70 task status = finished
>2012-12-17 08:54:31,182 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended, spm status: SPM
>2012-12-17 08:54:31,183 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e2ef5666-365a-4fac-8a08-89fcfd4b0a70), log id: 21a78d6
>2012-12-17 08:54:31,201 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMClearTaskVDSCommand, log id: 21a78d6
>2012-12-17 08:54:31,201 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@17f32aae, log id: 5692ab98
>2012-12-17 08:54:31,212 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:54:31,228 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:31,230 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5a1dfe90
>2012-12-17 08:54:31,248 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5a1dfe90
>2012-12-17 08:54:31,248 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7e595dac
>2012-12-17 08:54:31,248 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:54:31,248 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6733ce28
>2012-12-17 08:54:31,249 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:31,250 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 98ffb2c
>2012-12-17 08:54:31,267 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 98ffb2c
>2012-12-17 08:54:31,267 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6733ce28
>2012-12-17 08:54:31,267 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:54:37,987 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5266d504] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:54:40,153 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [6c95e244] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:54:40,458 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [192de1a2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:54:41,369 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6d27354c
>2012-12-17 08:54:41,371 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 51b2c770
>2012-12-17 08:54:41,391 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:41,585 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] FINISH, SpmStopVDSCommand, log id: 51b2c770
>2012-12-17 08:54:41,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] FINISH, ResetIrsVDSCommand, log id: 6d27354c
>2012-12-17 08:54:41,610 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ac82626
>2012-12-17 08:54:41,610 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [192de1a2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:41,611 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ac82626
>2012-12-17 08:54:48,182 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:54:51,666 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:54:51,676 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4689
>2012-12-17 08:54:51,677 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4689, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3efe533d
>2012-12-17 08:54:51,700 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] spmStart polling started: taskId = 91a11e87-ce54-4091-832c-f5e1a8f6c1cc
>2012-12-17 08:54:52,723 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] spmStart polling ended: taskId = 91a11e87-ce54-4091-832c-f5e1a8f6c1cc task status = finished
>2012-12-17 08:54:52,741 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] spmStart polling ended, spm status: SPM
>2012-12-17 08:54:52,742 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=91a11e87-ce54-4091-832c-f5e1a8f6c1cc), log id: 2afb0a2b
>2012-12-17 08:54:52,760 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, HSMClearTaskVDSCommand, log id: 2afb0a2b
>2012-12-17 08:54:52,761 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4758a318, log id: 3efe533d
>2012-12-17 08:54:52,774 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:54:52,803 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:54:52,803 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 8da5c0
>2012-12-17 08:54:52,837 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:54:52,947 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [46a80193] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:54:52,994 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3e581718
>2012-12-17 08:54:52,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 70b3167d
>2012-12-17 08:54:53,015 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [46a80193] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:53,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, SpmStopVDSCommand, log id: 70b3167d
>2012-12-17 08:54:53,717 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, ResetIrsVDSCommand, log id: 3e581718
>2012-12-17 08:54:53,729 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ba5d24f
>2012-12-17 08:54:53,729 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [46a80193] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:53,729 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ba5d24f
>2012-12-17 08:54:53,850 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:54:53,863 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4690
>2012-12-17 08:54:53,865 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4690, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 79eb4118
>2012-12-17 08:54:53,887 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling started: taskId = a3978184-ca31-4d15-bff9-a4e7512b5813
>2012-12-17 08:54:54,909 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended: taskId = a3978184-ca31-4d15-bff9-a4e7512b5813 task status = finished
>2012-12-17 08:54:54,928 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended, spm status: SPM
>2012-12-17 08:54:54,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a3978184-ca31-4d15-bff9-a4e7512b5813), log id: 8e5f177
>2012-12-17 08:54:54,948 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMClearTaskVDSCommand, log id: 8e5f177
>2012-12-17 08:54:54,948 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@304fc031, log id: 79eb4118
>2012-12-17 08:54:54,961 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:54:54,979 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:54,981 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: aac74de
>2012-12-17 08:54:54,999 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: aac74de
>2012-12-17 08:54:55,000 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 8da5c0
>2012-12-17 08:54:55,000 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:54:55,000 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 402ca04b
>2012-12-17 08:54:55,000 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:54:55,001 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 272c4b70
>2012-12-17 08:54:55,020 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 272c4b70
>2012-12-17 08:54:55,020 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 402ca04b
>2012-12-17 08:54:55,020 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:54:58,401 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:55:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [419474a0] Checking autorecoverable hosts
>2012-12-17 08:55:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [419474a0] Autorecovering 0 hosts
>2012-12-17 08:55:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [419474a0] Checking autorecoverable hosts done
>2012-12-17 08:55:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [419474a0] Autorecovering storage domains is disabled, skipping
>2012-12-17 08:55:03,918 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [79e50d62] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:55:04,043 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-75) [4dd32292] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:55:04,089 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4aa0d827
>2012-12-17 08:55:04,091 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2afb42d9
>2012-12-17 08:55:04,116 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:55:04,309 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, SpmStopVDSCommand, log id: 2afb42d9
>2012-12-17 08:55:04,324 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, ResetIrsVDSCommand, log id: 4aa0d827
>2012-12-17 08:55:04,335 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d4f1e7f
>2012-12-17 08:55:04,336 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:55:04,336 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d4f1e7f
>2012-12-17 08:55:08,600 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:55:14,385 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2ba7b70d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:55:14,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2ba7b70d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4691
>2012-12-17 08:55:14,410 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4691, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5f7ed5fe
>2012-12-17 08:55:14,431 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] spmStart polling started: taskId = 4b479d6a-8970-4261-a0a2-39543927e7a1
>2012-12-17 08:55:15,456 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] spmStart polling ended: taskId = 4b479d6a-8970-4261-a0a2-39543927e7a1 task status = finished
>2012-12-17 08:55:15,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] spmStart polling ended, spm status: SPM
>2012-12-17 08:55:15,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4b479d6a-8970-4261-a0a2-39543927e7a1), log id: 7aa7c379
>2012-12-17 08:55:15,495 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] FINISH, HSMClearTaskVDSCommand, log id: 7aa7c379
>2012-12-17 08:55:15,495 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c127bd8, log id: 5f7ed5fe
>2012-12-17 08:55:15,662 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2ba7b70d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:55:16,142 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [3d206f3c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:55:16,142 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 621299fa
>2012-12-17 08:55:16,157 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2ba7b70d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:55:16,291 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [4256c72b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:55:16,382 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [4256c72b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3a2f2df7
>2012-12-17 08:55:16,384 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [4256c72b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4687aeaf
>2012-12-17 08:55:16,403 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [4256c72b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:55:16,532 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [4256c72b] FINISH, SpmStopVDSCommand, log id: 4687aeaf
>2012-12-17 08:55:16,547 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [4256c72b] FINISH, ResetIrsVDSCommand, log id: 3a2f2df7
>2012-12-17 08:55:16,558 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [4256c72b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b094d46
>2012-12-17 08:55:16,558 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4256c72b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:55:16,558 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [4256c72b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b094d46
>2012-12-17 08:55:16,619 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3d206f3c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:55:16,649 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3d206f3c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4692
>2012-12-17 08:55:16,650 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4692, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5eceafa
>2012-12-17 08:55:16,673 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] spmStart polling started: taskId = 4d5ca2be-40f2-4c97-83c4-a84a1bcb2f76
>2012-12-17 08:55:17,697 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] spmStart polling ended: taskId = 4d5ca2be-40f2-4c97-83c4-a84a1bcb2f76 task status = finished
>2012-12-17 08:55:17,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] spmStart polling ended, spm status: SPM
>2012-12-17 08:55:17,717 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4d5ca2be-40f2-4c97-83c4-a84a1bcb2f76), log id: 2a8ff032
>2012-12-17 08:55:17,736 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, HSMClearTaskVDSCommand, log id: 2a8ff032
>2012-12-17 08:55:17,736 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4fe27872, log id: 5eceafa
>2012-12-17 08:55:17,749 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3d206f3c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:55:17,788 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:55:17,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4292e292
>2012-12-17 08:55:17,810 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4292e292
>2012-12-17 08:55:17,810 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 621299fa
>2012-12-17 08:55:17,810 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [3d206f3c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:55:17,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 39590163
>2012-12-17 08:55:17,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:55:17,812 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 197edd7
>2012-12-17 08:55:17,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 197edd7
>2012-12-17 08:55:17,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 39590163
>2012-12-17 08:55:17,830 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [3d206f3c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:55:18,888 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [5571991a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:55:26,707 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2f94bcb4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:55:26,878 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-12) [1407e3a2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:55:26,938 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 488bdd79
>2012-12-17 08:55:26,940 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 44968b88
>2012-12-17 08:55:26,961 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:55:27,126 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SpmStopVDSCommand, log id: 44968b88
>2012-12-17 08:55:27,141 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, ResetIrsVDSCommand, log id: 488bdd79
>2012-12-17 08:55:27,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5c458755
>2012-12-17 08:55:27,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:55:27,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5c458755
>2012-12-17 08:55:29,101 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [192de1a2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:55:37,214 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [34599a45] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:55:37,229 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [34599a45] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4693
>2012-12-17 08:55:37,230 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [34599a45] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4693, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 50813ea2
>2012-12-17 08:55:37,254 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [34599a45] spmStart polling started: taskId = 0ad7c0fc-a7c6-42bf-bb79-dff716f6a8d6
>2012-12-17 08:55:38,276 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [34599a45] spmStart polling ended: taskId = 0ad7c0fc-a7c6-42bf-bb79-dff716f6a8d6 task status = finished
>2012-12-17 08:55:38,294 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [34599a45] spmStart polling ended, spm status: SPM
>2012-12-17 08:55:38,296 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [34599a45] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0ad7c0fc-a7c6-42bf-bb79-dff716f6a8d6), log id: 17dc15a1
>2012-12-17 08:55:38,314 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [34599a45] FINISH, HSMClearTaskVDSCommand, log id: 17dc15a1
>2012-12-17 08:55:38,314 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [34599a45] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@72100943, log id: 50813ea2
>2012-12-17 08:55:38,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [34599a45] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:55:38,403 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:55:38,403 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2c854a17
>2012-12-17 08:55:38,421 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [34599a45] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:55:38,546 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [6cb72173] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:55:38,627 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [6cb72173] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17237abd
>2012-12-17 08:55:38,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [6cb72173] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5bbbd5b7
>2012-12-17 08:55:38,649 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [6cb72173] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:55:38,791 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [6cb72173] FINISH, SpmStopVDSCommand, log id: 5bbbd5b7
>2012-12-17 08:55:38,804 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [6cb72173] FINISH, ResetIrsVDSCommand, log id: 17237abd
>2012-12-17 08:55:38,815 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [6cb72173] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 61076859
>2012-12-17 08:55:38,815 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [6cb72173] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:55:38,815 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [6cb72173] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 61076859
>2012-12-17 08:55:38,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:55:38,920 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4694
>2012-12-17 08:55:38,922 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4694, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 557276fb
>2012-12-17 08:55:38,943 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling started: taskId = 8f8f7993-5b75-4af6-b7c0-143aa38adf73
>2012-12-17 08:55:39,968 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended: taskId = 8f8f7993-5b75-4af6-b7c0-143aa38adf73 task status = finished
>2012-12-17 08:55:39,985 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended, spm status: SPM
>2012-12-17 08:55:39,986 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8f8f7993-5b75-4af6-b7c0-143aa38adf73), log id: 404d51f7
>2012-12-17 08:55:40,004 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMClearTaskVDSCommand, log id: 404d51f7
>2012-12-17 08:55:40,004 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1440501c, log id: 557276fb
>2012-12-17 08:55:40,014 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:55:40,023 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:55:40,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5e168dc7
>2012-12-17 08:55:40,046 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5e168dc7
>2012-12-17 08:55:40,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2c854a17
>2012-12-17 08:55:40,046 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:55:40,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3014d918
>2012-12-17 08:55:40,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:55:40,048 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 123f2d66
>2012-12-17 08:55:40,066 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 123f2d66
>2012-12-17 08:55:40,066 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3014d918
>2012-12-17 08:55:40,066 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:55:48,896 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [dd34b5e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:55:48,997 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [48c4b2a7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:55:49,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [48c4b2a7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5480ff24
>2012-12-17 08:55:49,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [48c4b2a7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5b64ba6b
>2012-12-17 08:55:49,071 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [48c4b2a7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:55:49,225 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [48c4b2a7] FINISH, SpmStopVDSCommand, log id: 5b64ba6b
>2012-12-17 08:55:49,259 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [48c4b2a7] FINISH, ResetIrsVDSCommand, log id: 5480ff24
>2012-12-17 08:55:49,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [48c4b2a7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 721c04e2
>2012-12-17 08:55:49,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [48c4b2a7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:55:49,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [48c4b2a7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 721c04e2
>2012-12-17 08:55:49,564 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1a0f716] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:55:59,336 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [45590d60] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:55:59,349 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [45590d60] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4695
>2012-12-17 08:55:59,351 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [45590d60] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4695, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 468d743
>2012-12-17 08:55:59,372 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [45590d60] spmStart polling started: taskId = 020bf939-3f26-4c94-afcb-7cc000ec49f5
>2012-12-17 08:56:00,399 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [45590d60] spmStart polling ended: taskId = 020bf939-3f26-4c94-afcb-7cc000ec49f5 task status = finished
>2012-12-17 08:56:00,417 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [45590d60] spmStart polling ended, spm status: SPM
>2012-12-17 08:56:00,419 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [45590d60] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=020bf939-3f26-4c94-afcb-7cc000ec49f5), log id: 2e0139d1
>2012-12-17 08:56:00,438 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [45590d60] FINISH, HSMClearTaskVDSCommand, log id: 2e0139d1
>2012-12-17 08:56:00,438 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [45590d60] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@527adfef, log id: 468d743
>2012-12-17 08:56:00,698 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [45590d60] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:56:00,953 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [7b0d0943] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:56:00,954 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 15192f89
>2012-12-17 08:56:01,066 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [45590d60] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:56:01,381 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [4c00bcb5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:56:01,424 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [4c00bcb5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 153c787a
>2012-12-17 08:56:01,426 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [4c00bcb5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 455860cc
>2012-12-17 08:56:01,448 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [4c00bcb5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:01,880 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [4c00bcb5] FINISH, SpmStopVDSCommand, log id: 455860cc
>2012-12-17 08:56:01,898 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [4c00bcb5] FINISH, ResetIrsVDSCommand, log id: 153c787a
>2012-12-17 08:56:01,909 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [4c00bcb5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1599796e
>2012-12-17 08:56:01,909 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [4c00bcb5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:01,909 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [4c00bcb5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1599796e
>2012-12-17 08:56:01,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:56:02,007 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4696
>2012-12-17 08:56:02,009 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4696, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1f4d187d
>2012-12-17 08:56:02,031 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling started: taskId = 81716757-3b28-4230-85c3-2aaf9e86adcc
>2012-12-17 08:56:03,054 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling ended: taskId = 81716757-3b28-4230-85c3-2aaf9e86adcc task status = finished
>2012-12-17 08:56:03,075 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling ended, spm status: SPM
>2012-12-17 08:56:03,077 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=81716757-3b28-4230-85c3-2aaf9e86adcc), log id: 1c0d4bc8
>2012-12-17 08:56:03,100 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, HSMClearTaskVDSCommand, log id: 1c0d4bc8
>2012-12-17 08:56:03,100 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2cea45db, log id: 1f4d187d
>2012-12-17 08:56:03,137 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:56:03,155 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:03,157 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: dfdb312
>2012-12-17 08:56:03,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: dfdb312
>2012-12-17 08:56:03,178 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 15192f89
>2012-12-17 08:56:03,178 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [7b0d0943] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:56:03,178 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: af769eb
>2012-12-17 08:56:03,178 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:03,179 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7365931a
>2012-12-17 08:56:03,198 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7365931a
>2012-12-17 08:56:03,198 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: af769eb
>2012-12-17 08:56:03,198 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [7b0d0943] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:56:10,079 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [388f9500] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:56:12,021 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5266d504] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:56:12,135 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [46e0c818] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:56:12,178 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 74f44789
>2012-12-17 08:56:12,179 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26944008
>2012-12-17 08:56:12,199 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:12,352 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] FINISH, SpmStopVDSCommand, log id: 26944008
>2012-12-17 08:56:12,381 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] FINISH, ResetIrsVDSCommand, log id: 74f44789
>2012-12-17 08:56:12,392 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 79d20ae5
>2012-12-17 08:56:12,392 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [46e0c818] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:12,392 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 79d20ae5
>2012-12-17 08:56:20,313 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:56:22,468 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [44f1dfe] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:56:22,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [44f1dfe] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4697
>2012-12-17 08:56:22,481 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4697, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: c6b8b1
>2012-12-17 08:56:22,503 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] spmStart polling started: taskId = 9e48c035-7f0d-4f29-9aac-dffb46204705
>2012-12-17 08:56:23,526 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] spmStart polling ended: taskId = 9e48c035-7f0d-4f29-9aac-dffb46204705 task status = finished
>2012-12-17 08:56:23,545 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] spmStart polling ended, spm status: SPM
>2012-12-17 08:56:23,546 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9e48c035-7f0d-4f29-9aac-dffb46204705), log id: 4af6fe4b
>2012-12-17 08:56:23,565 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] FINISH, HSMClearTaskVDSCommand, log id: 4af6fe4b
>2012-12-17 08:56:23,565 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1db11460, log id: c6b8b1
>2012-12-17 08:56:23,577 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [44f1dfe] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:56:23,610 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [46701658] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:56:23,610 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5e1065b5
>2012-12-17 08:56:23,650 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [44f1dfe] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:56:23,798 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-44) [6fcdea84] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:56:23,877 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [6fcdea84] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2bdfc0d8
>2012-12-17 08:56:23,879 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [6fcdea84] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7fb1e469
>2012-12-17 08:56:23,899 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [6fcdea84] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:24,039 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [6fcdea84] FINISH, SpmStopVDSCommand, log id: 7fb1e469
>2012-12-17 08:56:24,072 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [6fcdea84] FINISH, ResetIrsVDSCommand, log id: 2bdfc0d8
>2012-12-17 08:56:24,083 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [6fcdea84] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6bb4df1d
>2012-12-17 08:56:24,084 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [6fcdea84] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:24,084 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [6fcdea84] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6bb4df1d
>2012-12-17 08:56:24,135 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:56:24,160 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4698
>2012-12-17 08:56:24,162 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4698, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 66ff6b15
>2012-12-17 08:56:24,184 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] spmStart polling started: taskId = c84f2dff-3b62-4c08-b300-074036761678
>2012-12-17 08:56:25,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] spmStart polling ended: taskId = c84f2dff-3b62-4c08-b300-074036761678 task status = finished
>2012-12-17 08:56:25,224 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] spmStart polling ended, spm status: SPM
>2012-12-17 08:56:25,226 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c84f2dff-3b62-4c08-b300-074036761678), log id: 37ae0b44
>2012-12-17 08:56:25,245 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, HSMClearTaskVDSCommand, log id: 37ae0b44
>2012-12-17 08:56:25,245 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@365b9eb, log id: 66ff6b15
>2012-12-17 08:56:25,292 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:56:25,302 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:25,303 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 206f0c74
>2012-12-17 08:56:25,325 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 206f0c74
>2012-12-17 08:56:25,325 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5e1065b5
>2012-12-17 08:56:25,325 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [46701658] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:56:25,325 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5a9aa44e
>2012-12-17 08:56:25,326 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:25,327 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6e6576fd
>2012-12-17 08:56:25,346 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6e6576fd
>2012-12-17 08:56:25,346 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5a9aa44e
>2012-12-17 08:56:25,346 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [46701658] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:56:31,256 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [1a6bc63] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:56:34,175 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1bff0c70] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:56:34,352 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [67ce2e5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:56:34,411 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 68010d29
>2012-12-17 08:56:34,413 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7c2feba
>2012-12-17 08:56:34,433 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:34,591 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] FINISH, SpmStopVDSCommand, log id: 7c2feba
>2012-12-17 08:56:34,613 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] FINISH, ResetIrsVDSCommand, log id: 68010d29
>2012-12-17 08:56:34,624 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c28a75f
>2012-12-17 08:56:34,624 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [67ce2e5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:34,624 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c28a75f
>2012-12-17 08:56:41,481 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:56:44,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [49c4851d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:56:44,721 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [49c4851d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4699
>2012-12-17 08:56:44,722 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4699, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 533c6115
>2012-12-17 08:56:44,744 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] spmStart polling started: taskId = 6b9cc4bf-8854-4ade-895b-de419f384d0e
>2012-12-17 08:56:45,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] spmStart polling ended: taskId = 6b9cc4bf-8854-4ade-895b-de419f384d0e task status = finished
>2012-12-17 08:56:45,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] spmStart polling ended, spm status: SPM
>2012-12-17 08:56:45,786 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6b9cc4bf-8854-4ade-895b-de419f384d0e), log id: 493dc77c
>2012-12-17 08:56:45,804 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] FINISH, HSMClearTaskVDSCommand, log id: 493dc77c
>2012-12-17 08:56:45,804 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@76f3c374, log id: 533c6115
>2012-12-17 08:56:46,020 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [49c4851d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:56:46,207 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:56:46,207 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2888347
>2012-12-17 08:56:46,464 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [49c4851d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:56:46,762 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [d0709d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:56:46,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 32d038ee
>2012-12-17 08:56:46,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6bdb6b00
>2012-12-17 08:56:46,849 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [d0709d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:46,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, SpmStopVDSCommand, log id: 6bdb6b00
>2012-12-17 08:56:47,001 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, ResetIrsVDSCommand, log id: 32d038ee
>2012-12-17 08:56:47,013 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5549cb05
>2012-12-17 08:56:47,013 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [d0709d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:47,013 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5549cb05
>2012-12-17 08:56:47,087 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:56:47,116 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4700
>2012-12-17 08:56:47,117 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4700, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: f463b21
>2012-12-17 08:56:47,139 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling started: taskId = 81baedd9-cb25-4130-a2d0-89e3980bcdb7
>2012-12-17 08:56:48,161 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended: taskId = 81baedd9-cb25-4130-a2d0-89e3980bcdb7 task status = finished
>2012-12-17 08:56:48,179 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended, spm status: SPM
>2012-12-17 08:56:48,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=81baedd9-cb25-4130-a2d0-89e3980bcdb7), log id: 3072e91b
>2012-12-17 08:56:48,199 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMClearTaskVDSCommand, log id: 3072e91b
>2012-12-17 08:56:48,199 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48260476, log id: f463b21
>2012-12-17 08:56:48,222 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:56:48,258 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:48,259 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5add7ddd
>2012-12-17 08:56:48,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5add7ddd
>2012-12-17 08:56:48,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2888347
>2012-12-17 08:56:48,281 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:56:48,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4bb952ee
>2012-12-17 08:56:48,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:48,283 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3900e32
>2012-12-17 08:56:48,301 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3900e32
>2012-12-17 08:56:48,301 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4bb952ee
>2012-12-17 08:56:48,301 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:56:51,705 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3d206f3c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:56:57,154 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [7c1d0790] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:56:57,297 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [356a6c1f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:56:57,415 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 212ba09c
>2012-12-17 08:56:57,416 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6bc5a3bf
>2012-12-17 08:56:57,436 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:58,185 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] FINISH, SpmStopVDSCommand, log id: 6bc5a3bf
>2012-12-17 08:56:58,245 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] FINISH, ResetIrsVDSCommand, log id: 212ba09c
>2012-12-17 08:56:58,256 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 22aec287
>2012-12-17 08:56:58,256 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [356a6c1f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:56:58,256 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 22aec287
>2012-12-17 08:57:01,922 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:57:08,327 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [44e11955] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:57:08,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [44e11955] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4701
>2012-12-17 08:57:08,358 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [44e11955] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4701, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5974d99a
>2012-12-17 08:57:08,380 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [44e11955] spmStart polling started: taskId = 71c8f917-b832-41a0-8d44-0747cc4b0e42
>2012-12-17 08:57:09,403 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [44e11955] spmStart polling ended: taskId = 71c8f917-b832-41a0-8d44-0747cc4b0e42 task status = finished
>2012-12-17 08:57:09,421 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [44e11955] spmStart polling ended, spm status: SPM
>2012-12-17 08:57:09,422 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [44e11955] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=71c8f917-b832-41a0-8d44-0747cc4b0e42), log id: 66a804f6
>2012-12-17 08:57:09,441 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [44e11955] FINISH, HSMClearTaskVDSCommand, log id: 66a804f6
>2012-12-17 08:57:09,441 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [44e11955] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@20dde721, log id: 5974d99a
>2012-12-17 08:57:09,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [44e11955] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:57:09,517 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [6d961413] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:57:09,517 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 442c314d
>2012-12-17 08:57:09,548 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [44e11955] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:57:09,713 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [1925ecb1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:57:09,769 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2103655f
>2012-12-17 08:57:09,771 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7a6e9afa
>2012-12-17 08:57:09,792 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:10,465 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, SpmStopVDSCommand, log id: 7a6e9afa
>2012-12-17 08:57:11,157 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, ResetIrsVDSCommand, log id: 2103655f
>2012-12-17 08:57:11,168 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5acb1f70
>2012-12-17 08:57:11,168 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:11,168 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5acb1f70
>2012-12-17 08:57:11,235 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [6d961413] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:57:11,248 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [6d961413] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4702
>2012-12-17 08:57:11,249 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4702, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2fd6a72b
>2012-12-17 08:57:11,271 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [6d961413] spmStart polling started: taskId = c3c21c65-5983-42b0-8d19-984b0b34ba3d
>2012-12-17 08:57:12,292 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [6d961413] spmStart polling ended: taskId = c3c21c65-5983-42b0-8d19-984b0b34ba3d task status = finished
>2012-12-17 08:57:12,310 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [6d961413] spmStart polling ended, spm status: SPM
>2012-12-17 08:57:12,312 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c3c21c65-5983-42b0-8d19-984b0b34ba3d), log id: 18b267ae
>2012-12-17 08:57:12,330 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, HSMClearTaskVDSCommand, log id: 18b267ae
>2012-12-17 08:57:12,330 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@68b4dd2c, log id: 2fd6a72b
>2012-12-17 08:57:12,342 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [6d961413] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:57:12,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:12,372 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3f0159bc
>2012-12-17 08:57:12,392 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3f0159bc
>2012-12-17 08:57:12,392 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 442c314d
>2012-12-17 08:57:12,392 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [6d961413] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:57:12,392 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6a0bb078
>2012-12-17 08:57:12,392 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:12,394 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 27704a28
>2012-12-17 08:57:12,413 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 27704a28
>2012-12-17 08:57:12,413 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6a0bb078
>2012-12-17 08:57:12,413 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [6d961413] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:57:21,292 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [48f6cd0b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:57:21,821 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [33e6033] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:57:21,883 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [33e6033] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 46946f56
>2012-12-17 08:57:21,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [33e6033] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1e3e7723
>2012-12-17 08:57:21,906 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [33e6033] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:22,080 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [33e6033] FINISH, SpmStopVDSCommand, log id: 1e3e7723
>2012-12-17 08:57:22,102 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [33e6033] FINISH, ResetIrsVDSCommand, log id: 46946f56
>2012-12-17 08:57:22,113 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [33e6033] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 51880028
>2012-12-17 08:57:22,113 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [33e6033] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:22,113 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [33e6033] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 51880028
>2012-12-17 08:57:22,358 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:57:32,166 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [4793062d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:57:32,183 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [4793062d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4703
>2012-12-17 08:57:32,184 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [4793062d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4703, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5c85f4c2
>2012-12-17 08:57:32,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [4793062d] spmStart polling started: taskId = 9f54f88b-beea-4f1c-b4ab-5c6d162285b8
>2012-12-17 08:57:33,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [4793062d] spmStart polling ended: taskId = 9f54f88b-beea-4f1c-b4ab-5c6d162285b8 task status = finished
>2012-12-17 08:57:33,246 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [4793062d] spmStart polling ended, spm status: SPM
>2012-12-17 08:57:33,248 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [4793062d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9f54f88b-beea-4f1c-b4ab-5c6d162285b8), log id: 752d01a5
>2012-12-17 08:57:33,267 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [4793062d] FINISH, HSMClearTaskVDSCommand, log id: 752d01a5
>2012-12-17 08:57:33,267 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [4793062d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5dc733bc, log id: 5c85f4c2
>2012-12-17 08:57:33,294 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [4793062d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:57:33,352 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [d0709d] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:57:33,352 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 184b53f6
>2012-12-17 08:57:33,363 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [4793062d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:57:33,518 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-36) [49b46a39] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:57:33,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [49b46a39] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 21972ad6
>2012-12-17 08:57:33,581 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [49b46a39] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 68f2910e
>2012-12-17 08:57:33,600 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [49b46a39] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:34,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [49b46a39] FINISH, SpmStopVDSCommand, log id: 68f2910e
>2012-12-17 08:57:34,538 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [49b46a39] FINISH, ResetIrsVDSCommand, log id: 21972ad6
>2012-12-17 08:57:34,549 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [49b46a39] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 39256663
>2012-12-17 08:57:34,549 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [49b46a39] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:34,549 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [49b46a39] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 39256663
>2012-12-17 08:57:34,625 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [d0709d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:57:34,685 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [d0709d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4704
>2012-12-17 08:57:34,687 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4704, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6bbd10cd
>2012-12-17 08:57:34,714 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [d0709d] spmStart polling started: taskId = b9e22d9e-606c-4671-a324-cf9bc3bbe76c
>2012-12-17 08:57:35,737 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [d0709d] spmStart polling ended: taskId = b9e22d9e-606c-4671-a324-cf9bc3bbe76c task status = finished
>2012-12-17 08:57:35,756 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [d0709d] spmStart polling ended, spm status: SPM
>2012-12-17 08:57:35,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b9e22d9e-606c-4671-a324-cf9bc3bbe76c), log id: 4607e6f5
>2012-12-17 08:57:35,776 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, HSMClearTaskVDSCommand, log id: 4607e6f5
>2012-12-17 08:57:35,776 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@30cc02be, log id: 6bbd10cd
>2012-12-17 08:57:36,157 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [d0709d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:57:36,200 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:36,202 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 31f9e0fb
>2012-12-17 08:57:36,223 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31f9e0fb
>2012-12-17 08:57:36,223 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 184b53f6
>2012-12-17 08:57:36,223 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [d0709d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:57:36,223 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 719a8b31
>2012-12-17 08:57:36,223 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:36,225 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 134a5c26
>2012-12-17 08:57:36,244 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 134a5c26
>2012-12-17 08:57:36,244 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 719a8b31
>2012-12-17 08:57:36,244 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [d0709d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:57:42,828 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:57:44,664 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:57:44,820 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-40) [346a0820] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:57:44,941 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6c7f8fbb
>2012-12-17 08:57:44,943 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4e313ef6
>2012-12-17 08:57:44,962 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-40) [346a0820] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:45,236 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SpmStopVDSCommand, log id: 4e313ef6
>2012-12-17 08:57:45,248 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, ResetIrsVDSCommand, log id: 6c7f8fbb
>2012-12-17 08:57:45,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6290a559
>2012-12-17 08:57:45,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:45,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6290a559
>2012-12-17 08:57:53,085 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [37630ba1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:57:55,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3969aa82] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:57:55,423 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3969aa82] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4705
>2012-12-17 08:57:55,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4705, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 328b6a4c
>2012-12-17 08:57:55,450 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] spmStart polling started: taskId = 5ca699a5-9352-4817-9c23-cb898ac3dc67
>2012-12-17 08:57:56,472 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] spmStart polling ended: taskId = 5ca699a5-9352-4817-9c23-cb898ac3dc67 task status = finished
>2012-12-17 08:57:56,490 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] spmStart polling ended, spm status: SPM
>2012-12-17 08:57:56,491 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5ca699a5-9352-4817-9c23-cb898ac3dc67), log id: 68b5d11d
>2012-12-17 08:57:56,511 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] FINISH, HSMClearTaskVDSCommand, log id: 68b5d11d
>2012-12-17 08:57:56,511 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78b3fc9f, log id: 328b6a4c
>2012-12-17 08:57:56,527 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3969aa82] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:57:56,620 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [3e5a5a03] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:57:56,621 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1b8228fe
>2012-12-17 08:57:56,656 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3969aa82] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:57:56,783 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [278943a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:57:56,844 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [278943a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2ef1926a
>2012-12-17 08:57:56,846 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [278943a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 742c0266
>2012-12-17 08:57:56,867 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [278943a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:57,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [278943a] FINISH, SpmStopVDSCommand, log id: 742c0266
>2012-12-17 08:57:57,804 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [278943a] FINISH, ResetIrsVDSCommand, log id: 2ef1926a
>2012-12-17 08:57:57,816 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [278943a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 749c520e
>2012-12-17 08:57:57,816 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [278943a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:57,816 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [278943a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 749c520e
>2012-12-17 08:57:57,876 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:57:57,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4706
>2012-12-17 08:57:57,896 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4706, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 402a76c8
>2012-12-17 08:57:57,917 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] spmStart polling started: taskId = 0b5da0be-b423-418a-9dbf-65414047a66b
>2012-12-17 08:57:58,938 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] spmStart polling ended: taskId = 0b5da0be-b423-418a-9dbf-65414047a66b task status = finished
>2012-12-17 08:57:58,957 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] spmStart polling ended, spm status: SPM
>2012-12-17 08:57:58,958 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0b5da0be-b423-418a-9dbf-65414047a66b), log id: 3bee9e6b
>2012-12-17 08:57:58,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, HSMClearTaskVDSCommand, log id: 3bee9e6b
>2012-12-17 08:57:58,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2ee2a18a, log id: 402a76c8
>2012-12-17 08:57:58,997 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:57:59,016 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:59,018 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7a42e8f
>2012-12-17 08:57:59,037 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7a42e8f
>2012-12-17 08:57:59,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1b8228fe
>2012-12-17 08:57:59,037 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [3e5a5a03] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:57:59,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 40a78a73
>2012-12-17 08:57:59,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:57:59,039 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: fa4347e
>2012-12-17 08:57:59,058 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: fa4347e
>2012-12-17 08:57:59,058 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 40a78a73
>2012-12-17 08:57:59,058 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [3e5a5a03] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:58:03,364 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:58:07,910 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [1a6bc63] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:58:08,035 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-56) [38fb24e7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:58:08,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 398e693a
>2012-12-17 08:58:08,096 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 18d3d562
>2012-12-17 08:58:08,116 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:58:08,338 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] FINISH, SpmStopVDSCommand, log id: 18d3d562
>2012-12-17 08:58:08,352 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] FINISH, ResetIrsVDSCommand, log id: 398e693a
>2012-12-17 08:58:08,364 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 34b3ad23
>2012-12-17 08:58:08,364 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [38fb24e7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:58:08,364 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 34b3ad23
>2012-12-17 08:58:13,728 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:58:18,425 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:58:18,453 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4707
>2012-12-17 08:58:18,455 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [1390e512] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4707, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3aa903b6
>2012-12-17 08:58:18,477 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [1390e512] spmStart polling started: taskId = 35c9239e-f779-41b2-bf0b-56599500ef00
>2012-12-17 08:58:19,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [1390e512] spmStart polling ended: taskId = 35c9239e-f779-41b2-bf0b-56599500ef00 task status = finished
>2012-12-17 08:58:19,519 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [1390e512] spmStart polling ended, spm status: SPM
>2012-12-17 08:58:19,520 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [1390e512] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=35c9239e-f779-41b2-bf0b-56599500ef00), log id: 372319bb
>2012-12-17 08:58:19,539 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [1390e512] FINISH, HSMClearTaskVDSCommand, log id: 372319bb
>2012-12-17 08:58:19,539 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [1390e512] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@616e6044, log id: 3aa903b6
>2012-12-17 08:58:19,619 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:58:19,684 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) [446aaffb] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:58:19,684 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6c3e0f43
>2012-12-17 08:58:19,779 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:58:20,014 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [22fa8e65] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:58:20,113 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3114a173
>2012-12-17 08:58:20,115 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 510b3404
>2012-12-17 08:58:20,135 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:58:20,270 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, SpmStopVDSCommand, log id: 510b3404
>2012-12-17 08:58:20,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, ResetIrsVDSCommand, log id: 3114a173
>2012-12-17 08:58:20,301 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7e0eecac
>2012-12-17 08:58:20,301 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [22fa8e65] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:58:20,301 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7e0eecac
>2012-12-17 08:58:20,505 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:58:20,964 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4708
>2012-12-17 08:58:20,966 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4708, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 669ada70
>2012-12-17 08:58:20,988 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] spmStart polling started: taskId = a6aae032-6170-4656-ae8a-a9e354b52342
>2012-12-17 08:58:22,011 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] spmStart polling ended: taskId = a6aae032-6170-4656-ae8a-a9e354b52342 task status = finished
>2012-12-17 08:58:22,029 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] spmStart polling ended, spm status: SPM
>2012-12-17 08:58:22,030 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a6aae032-6170-4656-ae8a-a9e354b52342), log id: 13a75b16
>2012-12-17 08:58:22,049 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, HSMClearTaskVDSCommand, log id: 13a75b16
>2012-12-17 08:58:22,049 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@504d0b6e, log id: 669ada70
>2012-12-17 08:58:22,065 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:58:22,110 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:58:22,112 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7aec2e08
>2012-12-17 08:58:22,139 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7aec2e08
>2012-12-17 08:58:22,139 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6c3e0f43
>2012-12-17 08:58:22,139 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) [446aaffb] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:58:22,139 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 723155fe
>2012-12-17 08:58:22,139 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:58:22,140 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1eba62be
>2012-12-17 08:58:22,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1eba62be
>2012-12-17 08:58:22,160 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 723155fe
>2012-12-17 08:58:22,160 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) [446aaffb] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:58:23,922 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [536655ff] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:58:30,836 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [5571991a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:58:31,125 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-28) [67d5cb38] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:58:31,273 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 678bb11a
>2012-12-17 08:58:31,275 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6f3a9a95
>2012-12-17 08:58:31,295 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:58:31,454 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] FINISH, SpmStopVDSCommand, log id: 6f3a9a95
>2012-12-17 08:58:31,512 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] FINISH, ResetIrsVDSCommand, log id: 678bb11a
>2012-12-17 08:58:31,522 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1b678a66
>2012-12-17 08:58:31,523 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [67d5cb38] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:58:31,523 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1b678a66
>2012-12-17 08:58:34,111 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [15ef2937] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:58:41,614 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [192de1a2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:58:41,650 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [192de1a2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4709
>2012-12-17 08:58:41,651 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4709, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 39265ae9
>2012-12-17 08:58:41,679 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] spmStart polling started: taskId = 9c0f610f-ff10-4028-865c-179a22ae4985
>2012-12-17 08:58:42,700 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] spmStart polling ended: taskId = 9c0f610f-ff10-4028-865c-179a22ae4985 task status = finished
>2012-12-17 08:58:42,718 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] spmStart polling ended, spm status: SPM
>2012-12-17 08:58:42,719 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9c0f610f-ff10-4028-865c-179a22ae4985), log id: 1b8f4e41
>2012-12-17 08:58:42,737 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] FINISH, HSMClearTaskVDSCommand, log id: 1b8f4e41
>2012-12-17 08:58:42,737 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d5a9b89, log id: 39265ae9
>2012-12-17 08:58:42,753 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [192de1a2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:58:42,827 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-52) [65048227] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:58:42,827 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2b71085f
>2012-12-17 08:58:42,874 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [192de1a2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:58:42,976 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [5044c757] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:58:43,040 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2986d40e
>2012-12-17 08:58:43,041 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4b554e9f
>2012-12-17 08:58:43,062 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [5044c757] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:58:43,660 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, SpmStopVDSCommand, log id: 4b554e9f
>2012-12-17 08:58:43,694 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, ResetIrsVDSCommand, log id: 2986d40e
>2012-12-17 08:58:43,704 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 26a5e912
>2012-12-17 08:58:43,704 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5044c757] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:58:43,704 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 26a5e912
>2012-12-17 08:58:43,806 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:58:43,837 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4710
>2012-12-17 08:58:43,839 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4710, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44422c2d
>2012-12-17 08:58:43,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] spmStart polling started: taskId = f16d4c87-3821-4fc9-aa0d-a7f07d9fb534
>2012-12-17 08:58:44,893 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] spmStart polling ended: taskId = f16d4c87-3821-4fc9-aa0d-a7f07d9fb534 task status = finished
>2012-12-17 08:58:44,911 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] spmStart polling ended, spm status: SPM
>2012-12-17 08:58:44,912 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f16d4c87-3821-4fc9-aa0d-a7f07d9fb534), log id: 386fc26b
>2012-12-17 08:58:44,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, HSMClearTaskVDSCommand, log id: 386fc26b
>2012-12-17 08:58:44,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31451bbf, log id: 44422c2d
>2012-12-17 08:58:44,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:58:44,961 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:58:44,963 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 13b4169f
>2012-12-17 08:58:44,984 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 13b4169f
>2012-12-17 08:58:44,984 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2b71085f
>2012-12-17 08:58:44,984 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-52) [65048227] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:58:44,984 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 352aceab
>2012-12-17 08:58:44,984 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:58:44,985 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 151549af
>2012-12-17 08:58:45,003 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 151549af
>2012-12-17 08:58:45,003 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 352aceab
>2012-12-17 08:58:45,003 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-52) [65048227] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:58:53,859 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5ad4c70] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:58:54,040 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [441eec2e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:58:54,191 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7e9ccd8a
>2012-12-17 08:58:54,193 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 767d2c88
>2012-12-17 08:58:54,212 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:58:54,483 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:58:54,529 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, SpmStopVDSCommand, log id: 767d2c88
>2012-12-17 08:58:54,556 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, ResetIrsVDSCommand, log id: 7e9ccd8a
>2012-12-17 08:58:54,566 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d211c63
>2012-12-17 08:58:54,566 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:58:54,566 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d211c63
>2012-12-17 08:59:04,632 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [164629b0] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:59:04,649 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [164629b0] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4711
>2012-12-17 08:59:04,651 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [164629b0] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4711, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40395881
>2012-12-17 08:59:04,673 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [164629b0] spmStart polling started: taskId = bd9f33d7-b55c-4013-b288-9313a0f27bfc
>2012-12-17 08:59:05,696 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [164629b0] spmStart polling ended: taskId = bd9f33d7-b55c-4013-b288-9313a0f27bfc task status = finished
>2012-12-17 08:59:05,715 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [164629b0] spmStart polling ended, spm status: SPM
>2012-12-17 08:59:05,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [164629b0] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bd9f33d7-b55c-4013-b288-9313a0f27bfc), log id: 4dd19dce
>2012-12-17 08:59:05,736 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [164629b0] FINISH, HSMClearTaskVDSCommand, log id: 4dd19dce
>2012-12-17 08:59:05,736 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [164629b0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7ccf674b, log id: 40395881
>2012-12-17 08:59:06,029 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [164629b0] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:59:06,103 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:59:06,104 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 107cc8c0
>2012-12-17 08:59:06,117 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [164629b0] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:59:06,308 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [71b7bdef] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:59:06,382 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 247a2196
>2012-12-17 08:59:06,384 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3984a49b
>2012-12-17 08:59:06,404 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:06,675 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, SpmStopVDSCommand, log id: 3984a49b
>2012-12-17 08:59:06,691 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, ResetIrsVDSCommand, log id: 247a2196
>2012-12-17 08:59:06,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: fb5e0f9
>2012-12-17 08:59:06,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:06,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, MarkPoolInReconstructModeVDSCommand, log id: fb5e0f9
>2012-12-17 08:59:06,755 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:59:06,762 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4712
>2012-12-17 08:59:06,764 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4712, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 28616981
>2012-12-17 08:59:06,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling started: taskId = 3b310586-cac4-468a-a124-fe2262d64538
>2012-12-17 08:59:07,811 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended: taskId = 3b310586-cac4-468a-a124-fe2262d64538 task status = finished
>2012-12-17 08:59:07,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended, spm status: SPM
>2012-12-17 08:59:07,832 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3b310586-cac4-468a-a124-fe2262d64538), log id: 4e1bcc8b
>2012-12-17 08:59:07,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMClearTaskVDSCommand, log id: 4e1bcc8b
>2012-12-17 08:59:07,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@311e1bd, log id: 28616981
>2012-12-17 08:59:07,867 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:59:07,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:07,896 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 11d522b7
>2012-12-17 08:59:07,915 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 11d522b7
>2012-12-17 08:59:07,915 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 107cc8c0
>2012-12-17 08:59:07,915 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:59:07,915 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3fda2586
>2012-12-17 08:59:07,915 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:07,916 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3d809042
>2012-12-17 08:59:07,935 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3d809042
>2012-12-17 08:59:07,936 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3fda2586
>2012-12-17 08:59:07,936 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:59:14,887 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [5a014659] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:59:16,803 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3d206f3c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:59:16,945 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [a029edc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:59:17,023 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 83d0ca7
>2012-12-17 08:59:17,026 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e2fe42d
>2012-12-17 08:59:17,056 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [a029edc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:17,272 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, SpmStopVDSCommand, log id: 7e2fe42d
>2012-12-17 08:59:17,291 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, ResetIrsVDSCommand, log id: 83d0ca7
>2012-12-17 08:59:17,302 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ff37726
>2012-12-17 08:59:17,302 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:17,302 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ff37726
>2012-12-17 08:59:25,079 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:59:27,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [33e4b3f5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:59:27,424 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [33e4b3f5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4713
>2012-12-17 08:59:27,425 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4713, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3aa6adc2
>2012-12-17 08:59:27,447 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] spmStart polling started: taskId = 92d6899e-daea-4dee-9e1c-d06c90d33c36
>2012-12-17 08:59:28,470 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] spmStart polling ended: taskId = 92d6899e-daea-4dee-9e1c-d06c90d33c36 task status = finished
>2012-12-17 08:59:28,488 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] spmStart polling ended, spm status: SPM
>2012-12-17 08:59:28,490 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=92d6899e-daea-4dee-9e1c-d06c90d33c36), log id: 7d8a9dcb
>2012-12-17 08:59:28,508 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] FINISH, HSMClearTaskVDSCommand, log id: 7d8a9dcb
>2012-12-17 08:59:28,508 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@177dbad1, log id: 3aa6adc2
>2012-12-17 08:59:28,554 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [33e4b3f5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:59:28,662 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [61136831] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:59:28,662 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7a928470
>2012-12-17 08:59:28,662 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [33e4b3f5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:59:28,816 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [39739d98] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:59:28,868 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 76d00446
>2012-12-17 08:59:28,870 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 51ced073
>2012-12-17 08:59:28,890 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [39739d98] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:29,413 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, SpmStopVDSCommand, log id: 51ced073
>2012-12-17 08:59:29,423 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, ResetIrsVDSCommand, log id: 76d00446
>2012-12-17 08:59:29,434 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4479c005
>2012-12-17 08:59:29,434 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:29,434 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4479c005
>2012-12-17 08:59:29,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:59:29,510 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4714
>2012-12-17 08:59:29,512 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4714, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5420893f
>2012-12-17 08:59:29,536 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] spmStart polling started: taskId = ee4c5cc7-eeeb-4e2a-9b29-d8bc504431a3
>2012-12-17 08:59:30,558 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] spmStart polling ended: taskId = ee4c5cc7-eeeb-4e2a-9b29-d8bc504431a3 task status = finished
>2012-12-17 08:59:30,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] spmStart polling ended, spm status: SPM
>2012-12-17 08:59:30,577 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ee4c5cc7-eeeb-4e2a-9b29-d8bc504431a3), log id: 45e3ce5f
>2012-12-17 08:59:30,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, HSMClearTaskVDSCommand, log id: 45e3ce5f
>2012-12-17 08:59:30,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6f01e2d1, log id: 5420893f
>2012-12-17 08:59:31,187 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:59:31,219 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:31,221 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3f037e02
>2012-12-17 08:59:31,243 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3f037e02
>2012-12-17 08:59:31,243 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7a928470
>2012-12-17 08:59:31,243 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [61136831] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:59:31,243 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 507c5f4f
>2012-12-17 08:59:31,243 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:31,244 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3bcd3ff7
>2012-12-17 08:59:31,263 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3bcd3ff7
>2012-12-17 08:59:31,263 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 507c5f4f
>2012-12-17 08:59:31,263 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [61136831] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:59:35,316 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [4557a90c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:59:39,544 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:59:39,667 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [53d0bc2b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:59:39,710 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [53d0bc2b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: a8e1387
>2012-12-17 08:59:39,712 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [53d0bc2b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7719dc70
>2012-12-17 08:59:39,732 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [53d0bc2b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:39,879 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [53d0bc2b] FINISH, SpmStopVDSCommand, log id: 7719dc70
>2012-12-17 08:59:39,904 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [53d0bc2b] FINISH, ResetIrsVDSCommand, log id: a8e1387
>2012-12-17 08:59:39,915 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [53d0bc2b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c00db98
>2012-12-17 08:59:39,915 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [53d0bc2b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:39,915 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [53d0bc2b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c00db98
>2012-12-17 08:59:46,824 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 08:59:49,986 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:59:50,000 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4715
>2012-12-17 08:59:50,001 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4715, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 50e10840
>2012-12-17 08:59:50,026 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling started: taskId = eec005ab-96df-40ce-95d6-a56f3ab45491
>2012-12-17 08:59:51,047 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling ended: taskId = eec005ab-96df-40ce-95d6-a56f3ab45491 task status = finished
>2012-12-17 08:59:51,065 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling ended, spm status: SPM
>2012-12-17 08:59:51,066 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=eec005ab-96df-40ce-95d6-a56f3ab45491), log id: 6aeb99cc
>2012-12-17 08:59:51,085 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, HSMClearTaskVDSCommand, log id: 6aeb99cc
>2012-12-17 08:59:51,085 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@11be56eb, log id: 50e10840
>2012-12-17 08:59:51,115 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:59:51,151 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-42) [27a8ed97] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 08:59:51,151 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 135619dd
>2012-12-17 08:59:51,167 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 08:59:51,251 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [579f6ed1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 08:59:51,317 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [579f6ed1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 729aec05
>2012-12-17 08:59:51,319 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [579f6ed1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 233aacaf
>2012-12-17 08:59:51,338 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [579f6ed1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:51,837 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [579f6ed1] FINISH, SpmStopVDSCommand, log id: 233aacaf
>2012-12-17 08:59:51,849 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [579f6ed1] FINISH, ResetIrsVDSCommand, log id: 729aec05
>2012-12-17 08:59:51,860 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [579f6ed1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7b5956c0
>2012-12-17 08:59:51,860 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [579f6ed1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:51,860 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [579f6ed1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7b5956c0
>2012-12-17 08:59:51,924 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 08:59:51,940 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4716
>2012-12-17 08:59:51,941 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4716, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b3c099c
>2012-12-17 08:59:51,964 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] spmStart polling started: taskId = 1619d4d3-e156-4798-80d0-7a178c49dc16
>2012-12-17 08:59:52,985 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] spmStart polling ended: taskId = 1619d4d3-e156-4798-80d0-7a178c49dc16 task status = finished
>2012-12-17 08:59:53,003 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] spmStart polling ended, spm status: SPM
>2012-12-17 08:59:53,005 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1619d4d3-e156-4798-80d0-7a178c49dc16), log id: f9c7ef8
>2012-12-17 08:59:53,023 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, HSMClearTaskVDSCommand, log id: f9c7ef8
>2012-12-17 08:59:53,023 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6b53d3c9, log id: 6b3c099c
>2012-12-17 08:59:53,040 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 08:59:53,061 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:53,066 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 78ff06fa
>2012-12-17 08:59:53,085 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 78ff06fa
>2012-12-17 08:59:53,085 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 135619dd
>2012-12-17 08:59:53,085 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-42) [27a8ed97] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:59:53,085 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2f234525
>2012-12-17 08:59:53,085 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 08:59:53,086 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 119fc441
>2012-12-17 08:59:53,105 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 119fc441
>2012-12-17 08:59:53,105 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2f234525
>2012-12-17 08:59:53,105 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-42) [27a8ed97] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 08:59:57,059 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5abe3acd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:00:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [6bb29080] Checking autorecoverable hosts
>2012-12-17 09:00:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [6bb29080] Autorecovering 0 hosts
>2012-12-17 09:00:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [6bb29080] Checking autorecoverable hosts done
>2012-12-17 09:00:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [6bb29080] Autorecovering storage domains is disabled, skipping
>2012-12-17 09:00:01,966 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [5a014659] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:00:02,099 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-18) [7790038] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:00:02,142 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4bc1c8fc
>2012-12-17 09:00:02,152 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: b0a0f69
>2012-12-17 09:00:02,175 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [7790038] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:00:02,314 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, SpmStopVDSCommand, log id: b0a0f69
>2012-12-17 09:00:02,342 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, ResetIrsVDSCommand, log id: 4bc1c8fc
>2012-12-17 09:00:02,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3e7f4a72
>2012-12-17 09:00:02,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:00:02,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3e7f4a72
>2012-12-17 09:00:07,267 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [5231d6f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:00:12,416 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:00:12,445 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4717
>2012-12-17 09:00:12,447 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4717, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7c59ba03
>2012-12-17 09:00:12,470 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] spmStart polling started: taskId = a7888054-b858-429e-a546-d5fb66202c8f
>2012-12-17 09:00:13,490 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] spmStart polling ended: taskId = a7888054-b858-429e-a546-d5fb66202c8f task status = finished
>2012-12-17 09:00:13,508 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] spmStart polling ended, spm status: SPM
>2012-12-17 09:00:13,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a7888054-b858-429e-a546-d5fb66202c8f), log id: fe4c4c5
>2012-12-17 09:00:13,527 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, HSMClearTaskVDSCommand, log id: fe4c4c5
>2012-12-17 09:00:13,527 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@743269a9, log id: 7c59ba03
>2012-12-17 09:00:13,541 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:00:13,577 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [5ab536e] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:00:13,577 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 517b692
>2012-12-17 09:00:13,597 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:00:13,688 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [680d007a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:00:13,760 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [680d007a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 63909f88
>2012-12-17 09:00:13,762 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [680d007a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 23d36a1b
>2012-12-17 09:00:13,781 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [680d007a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:00:13,919 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [680d007a] FINISH, SpmStopVDSCommand, log id: 23d36a1b
>2012-12-17 09:00:13,936 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [680d007a] FINISH, ResetIrsVDSCommand, log id: 63909f88
>2012-12-17 09:00:13,947 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [680d007a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 241b9e33
>2012-12-17 09:00:13,947 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [680d007a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:00:13,947 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [680d007a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 241b9e33
>2012-12-17 09:00:14,008 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:00:14,019 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4718
>2012-12-17 09:00:14,020 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4718, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2edbbe5e
>2012-12-17 09:00:14,043 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] spmStart polling started: taskId = 0d6106c8-0b6d-4ff3-be15-cbe6ca1a3b80
>2012-12-17 09:00:15,066 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] spmStart polling ended: taskId = 0d6106c8-0b6d-4ff3-be15-cbe6ca1a3b80 task status = finished
>2012-12-17 09:00:15,085 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] spmStart polling ended, spm status: SPM
>2012-12-17 09:00:15,086 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0d6106c8-0b6d-4ff3-be15-cbe6ca1a3b80), log id: 6cdb0cac
>2012-12-17 09:00:15,105 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, HSMClearTaskVDSCommand, log id: 6cdb0cac
>2012-12-17 09:00:15,105 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@211d2809, log id: 2edbbe5e
>2012-12-17 09:00:15,118 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:00:15,127 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:00:15,131 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 851b771
>2012-12-17 09:00:15,150 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 851b771
>2012-12-17 09:00:15,150 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 517b692
>2012-12-17 09:00:15,150 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [5ab536e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:00:15,150 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 420b6ec2
>2012-12-17 09:00:15,150 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:00:15,151 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3441c536
>2012-12-17 09:00:15,170 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3441c536
>2012-12-17 09:00:15,170 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 420b6ec2
>2012-12-17 09:00:15,170 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [5ab536e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:00:17,544 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:00:24,050 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [6d961413] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:00:24,153 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [7781d588] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:00:24,216 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [7781d588] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6286e2a6
>2012-12-17 09:00:24,218 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [7781d588] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 63d935fe
>2012-12-17 09:00:24,237 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [7781d588] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:00:24,377 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [7781d588] FINISH, SpmStopVDSCommand, log id: 63d935fe
>2012-12-17 09:00:24,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [7781d588] FINISH, ResetIrsVDSCommand, log id: 6286e2a6
>2012-12-17 09:00:24,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [7781d588] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 994de38
>2012-12-17 09:00:24,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [7781d588] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:00:24,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [7781d588] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 994de38
>2012-12-17 09:00:27,752 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:00:34,466 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1a0f716] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:00:34,479 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1a0f716] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4719
>2012-12-17 09:00:34,481 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4719, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7cde0a5d
>2012-12-17 09:00:34,505 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] spmStart polling started: taskId = a246e9df-90a1-4272-af66-ad6cc984bd70
>2012-12-17 09:00:35,527 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] spmStart polling ended: taskId = a246e9df-90a1-4272-af66-ad6cc984bd70 task status = finished
>2012-12-17 09:00:35,546 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] spmStart polling ended, spm status: SPM
>2012-12-17 09:00:35,547 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a246e9df-90a1-4272-af66-ad6cc984bd70), log id: 13b3e0dd
>2012-12-17 09:00:35,565 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] FINISH, HSMClearTaskVDSCommand, log id: 13b3e0dd
>2012-12-17 09:00:35,565 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@66fd746, log id: 7cde0a5d
>2012-12-17 09:00:35,786 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1a0f716] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:00:36,569 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:00:36,569 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 11c49f61
>2012-12-17 09:00:36,714 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1a0f716] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:00:36,931 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:00:36,985 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4cf0fb7a
>2012-12-17 09:00:36,987 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 201f3121
>2012-12-17 09:00:37,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:00:37,126 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] FINISH, SpmStopVDSCommand, log id: 201f3121
>2012-12-17 09:00:37,141 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] FINISH, ResetIrsVDSCommand, log id: 4cf0fb7a
>2012-12-17 09:00:37,153 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1d489bac
>2012-12-17 09:00:37,153 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:00:37,153 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1d489bac
>2012-12-17 09:00:37,217 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:00:37,242 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4720
>2012-12-17 09:00:37,244 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4720, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 16895bbb
>2012-12-17 09:00:37,266 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling started: taskId = f9357a86-f6a5-48c1-a89b-bc5896da3391
>2012-12-17 09:00:38,287 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling ended: taskId = f9357a86-f6a5-48c1-a89b-bc5896da3391 task status = finished
>2012-12-17 09:00:38,306 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling ended, spm status: SPM
>2012-12-17 09:00:38,307 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f9357a86-f6a5-48c1-a89b-bc5896da3391), log id: 26103050
>2012-12-17 09:00:38,325 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMClearTaskVDSCommand, log id: 26103050
>2012-12-17 09:00:38,325 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@70d3e7b5, log id: 16895bbb
>2012-12-17 09:00:38,337 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:00:38,352 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:00:38,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: ec27655
>2012-12-17 09:00:38,374 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: ec27655
>2012-12-17 09:00:38,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 11c49f61
>2012-12-17 09:00:38,374 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:00:38,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 30dfaab9
>2012-12-17 09:00:38,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:00:38,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d2875e5
>2012-12-17 09:00:38,394 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d2875e5
>2012-12-17 09:00:38,394 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 30dfaab9
>2012-12-17 09:00:38,394 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:00:47,360 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [536655ff] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:00:47,581 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-49) [30604458] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:00:47,652 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [30604458] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 45d783f1
>2012-12-17 09:00:47,654 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [30604458] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 19ab309f
>2012-12-17 09:00:47,673 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [30604458] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:00:48,207 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [4c00bcb5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:00:48,440 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [30604458] FINISH, SpmStopVDSCommand, log id: 19ab309f
>2012-12-17 09:00:48,472 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [30604458] FINISH, ResetIrsVDSCommand, log id: 45d783f1
>2012-12-17 09:00:48,483 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [30604458] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 280d1527
>2012-12-17 09:00:48,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [30604458] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:00:48,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [30604458] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 280d1527
>2012-12-17 09:00:58,548 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:00:58,580 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4721
>2012-12-17 09:00:58,581 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4721, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67382da
>2012-12-17 09:00:58,603 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] spmStart polling started: taskId = 9a64b5b1-0cb4-44bf-a0b3-14ac2a250c2f
>2012-12-17 09:00:59,625 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] spmStart polling ended: taskId = 9a64b5b1-0cb4-44bf-a0b3-14ac2a250c2f task status = finished
>2012-12-17 09:00:59,643 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] spmStart polling ended, spm status: SPM
>2012-12-17 09:00:59,645 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9a64b5b1-0cb4-44bf-a0b3-14ac2a250c2f), log id: 27e66ecd
>2012-12-17 09:00:59,663 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] FINISH, HSMClearTaskVDSCommand, log id: 27e66ecd
>2012-12-17 09:00:59,663 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@9746932, log id: 67382da
>2012-12-17 09:00:59,678 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:00:59,713 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:00:59,713 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 15f85a76
>2012-12-17 09:00:59,735 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:00:59,842 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-29) [50b67ada] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:00:59,884 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [50b67ada] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 185f49a6
>2012-12-17 09:00:59,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [50b67ada] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 16630bec
>2012-12-17 09:00:59,906 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [50b67ada] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:00,071 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [50b67ada] FINISH, SpmStopVDSCommand, log id: 16630bec
>2012-12-17 09:01:00,127 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [50b67ada] FINISH, ResetIrsVDSCommand, log id: 185f49a6
>2012-12-17 09:01:00,177 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [50b67ada] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6d85f31a
>2012-12-17 09:01:00,177 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [50b67ada] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:00,177 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [50b67ada] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6d85f31a
>2012-12-17 09:01:00,269 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:01:00,297 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4722
>2012-12-17 09:01:00,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4722, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6772d648
>2012-12-17 09:01:00,320 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling started: taskId = 19050ad5-45d2-4e6e-9ece-2319ec223c12
>2012-12-17 09:01:01,343 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended: taskId = 19050ad5-45d2-4e6e-9ece-2319ec223c12 task status = finished
>2012-12-17 09:01:01,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended, spm status: SPM
>2012-12-17 09:01:01,363 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=19050ad5-45d2-4e6e-9ece-2319ec223c12), log id: 7be95813
>2012-12-17 09:01:01,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMClearTaskVDSCommand, log id: 7be95813
>2012-12-17 09:01:01,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@149586f3, log id: 6772d648
>2012-12-17 09:01:01,397 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:01:01,422 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:01,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 49ba570
>2012-12-17 09:01:01,442 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 49ba570
>2012-12-17 09:01:01,443 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 15f85a76
>2012-12-17 09:01:01,443 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:01:01,443 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 663d4975
>2012-12-17 09:01:01,443 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:01,444 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2b8fe719
>2012-12-17 09:01:01,463 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2b8fe719
>2012-12-17 09:01:01,463 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 663d4975
>2012-12-17 09:01:01,463 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:01:08,674 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:01:10,363 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:01:11,334 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-81) [4fb47a41] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:01:11,529 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3afb4c29
>2012-12-17 09:01:11,531 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 650e8b48
>2012-12-17 09:01:11,551 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:11,749 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, SpmStopVDSCommand, log id: 650e8b48
>2012-12-17 09:01:11,768 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, ResetIrsVDSCommand, log id: 3afb4c29
>2012-12-17 09:01:11,779 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ddd8111
>2012-12-17 09:01:11,779 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:11,779 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ddd8111
>2012-12-17 09:01:18,900 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:01:21,832 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [67ce2e5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:01:21,844 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [67ce2e5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4723
>2012-12-17 09:01:21,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4723, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 65d0e230
>2012-12-17 09:01:21,868 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] spmStart polling started: taskId = 909b1e66-2d4b-4293-b99d-4682c47ba929
>2012-12-17 09:01:22,893 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] spmStart polling ended: taskId = 909b1e66-2d4b-4293-b99d-4682c47ba929 task status = finished
>2012-12-17 09:01:22,913 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] spmStart polling ended, spm status: SPM
>2012-12-17 09:01:22,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=909b1e66-2d4b-4293-b99d-4682c47ba929), log id: eaf1bcc
>2012-12-17 09:01:22,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] FINISH, HSMClearTaskVDSCommand, log id: eaf1bcc
>2012-12-17 09:01:22,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@a49f59c, log id: 65d0e230
>2012-12-17 09:01:22,940 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [67ce2e5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:01:22,991 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5bc809fe] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:01:22,992 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 36f818e6
>2012-12-17 09:01:23,021 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [67ce2e5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:01:23,133 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [32d89e5e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:01:23,200 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 54379061
>2012-12-17 09:01:23,202 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2d120fea
>2012-12-17 09:01:23,221 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:24,006 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, SpmStopVDSCommand, log id: 2d120fea
>2012-12-17 09:01:24,026 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, ResetIrsVDSCommand, log id: 54379061
>2012-12-17 09:01:24,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 590012ac
>2012-12-17 09:01:24,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [32d89e5e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:24,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 590012ac
>2012-12-17 09:01:24,169 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:01:24,177 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4724
>2012-12-17 09:01:24,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4724, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2e187dc8
>2012-12-17 09:01:24,201 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] spmStart polling started: taskId = 5483c80b-689a-4de1-ad4e-e136cdbdb812
>2012-12-17 09:01:25,221 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] spmStart polling ended: taskId = 5483c80b-689a-4de1-ad4e-e136cdbdb812 task status = finished
>2012-12-17 09:01:25,240 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] spmStart polling ended, spm status: SPM
>2012-12-17 09:01:25,241 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5483c80b-689a-4de1-ad4e-e136cdbdb812), log id: 3535176a
>2012-12-17 09:01:25,259 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, HSMClearTaskVDSCommand, log id: 3535176a
>2012-12-17 09:01:25,259 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@30944b70, log id: 2e187dc8
>2012-12-17 09:01:25,285 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:01:25,337 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:25,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6c61f68d
>2012-12-17 09:01:25,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6c61f68d
>2012-12-17 09:01:25,360 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 36f818e6
>2012-12-17 09:01:25,360 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5bc809fe] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:01:25,360 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1381da92
>2012-12-17 09:01:25,360 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:25,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d44894d
>2012-12-17 09:01:25,380 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d44894d
>2012-12-17 09:01:25,380 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1381da92
>2012-12-17 09:01:25,380 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5bc809fe] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:01:29,118 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:01:34,224 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [d0709d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:01:34,395 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [25c7cf42] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:01:34,445 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 27a929a8
>2012-12-17 09:01:34,446 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6c40bc14
>2012-12-17 09:01:34,477 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:34,666 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, SpmStopVDSCommand, log id: 6c40bc14
>2012-12-17 09:01:34,683 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, ResetIrsVDSCommand, log id: 27a929a8
>2012-12-17 09:01:34,694 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 763b5eb0
>2012-12-17 09:01:34,694 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [25c7cf42] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:34,694 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 763b5eb0
>2012-12-17 09:01:39,366 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [15510694] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:01:44,766 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3ef697d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:01:44,784 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3ef697d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4725
>2012-12-17 09:01:44,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4725, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 36cbc8f
>2012-12-17 09:01:44,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] spmStart polling started: taskId = 95a0afad-bb7b-446c-afe0-51628a88a307
>2012-12-17 09:01:45,832 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] spmStart polling ended: taskId = 95a0afad-bb7b-446c-afe0-51628a88a307 task status = finished
>2012-12-17 09:01:45,850 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] spmStart polling ended, spm status: SPM
>2012-12-17 09:01:45,852 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=95a0afad-bb7b-446c-afe0-51628a88a307), log id: 4336a12f
>2012-12-17 09:01:45,870 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] FINISH, HSMClearTaskVDSCommand, log id: 4336a12f
>2012-12-17 09:01:45,870 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@37903722, log id: 36cbc8f
>2012-12-17 09:01:45,944 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3ef697d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:01:46,190 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:01:46,190 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 11ebf61a
>2012-12-17 09:01:46,212 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3ef697d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:01:46,425 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [19e66a45] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:01:46,485 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 49a934b9
>2012-12-17 09:01:46,486 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2b8dafe6
>2012-12-17 09:01:46,506 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:46,634 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] FINISH, SpmStopVDSCommand, log id: 2b8dafe6
>2012-12-17 09:01:46,675 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] FINISH, ResetIrsVDSCommand, log id: 49a934b9
>2012-12-17 09:01:46,686 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 799e162b
>2012-12-17 09:01:46,686 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [19e66a45] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:46,686 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 799e162b
>2012-12-17 09:01:46,755 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:01:46,767 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4726
>2012-12-17 09:01:46,768 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4726, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4fad596e
>2012-12-17 09:01:46,791 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling started: taskId = 597723ef-9024-403b-b691-7daaa3684fa5
>2012-12-17 09:01:47,813 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling ended: taskId = 597723ef-9024-403b-b691-7daaa3684fa5 task status = finished
>2012-12-17 09:01:47,831 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling ended, spm status: SPM
>2012-12-17 09:01:47,832 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=597723ef-9024-403b-b691-7daaa3684fa5), log id: b6a5fb3
>2012-12-17 09:01:47,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMClearTaskVDSCommand, log id: b6a5fb3
>2012-12-17 09:01:47,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@74b0e69f, log id: 4fad596e
>2012-12-17 09:01:47,858 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:01:47,898 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:47,903 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3e586a35
>2012-12-17 09:01:47,923 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3e586a35
>2012-12-17 09:01:47,923 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 11ebf61a
>2012-12-17 09:01:47,923 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:01:47,923 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1ea4dd8d
>2012-12-17 09:01:47,923 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:47,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 797adcb9
>2012-12-17 09:01:47,942 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 797adcb9
>2012-12-17 09:01:47,942 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1ea4dd8d
>2012-12-17 09:01:47,942 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:01:49,606 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [37630ba1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:01:56,794 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [6cb72173] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:01:56,882 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [1c12d57c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:01:56,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [1c12d57c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 22f9919a
>2012-12-17 09:01:56,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [1c12d57c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 223b5ba4
>2012-12-17 09:01:56,953 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [1c12d57c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:57,119 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [1c12d57c] FINISH, SpmStopVDSCommand, log id: 223b5ba4
>2012-12-17 09:01:57,134 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [1c12d57c] FINISH, ResetIrsVDSCommand, log id: 22f9919a
>2012-12-17 09:01:57,145 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [1c12d57c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ae0b5bc
>2012-12-17 09:01:57,145 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [1c12d57c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:01:57,145 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [1c12d57c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ae0b5bc
>2012-12-17 09:01:59,799 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:02:07,198 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [32c2c5a9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:02:07,212 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [32c2c5a9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4727
>2012-12-17 09:02:07,213 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4727, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 45954737
>2012-12-17 09:02:07,236 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] spmStart polling started: taskId = c67ed1ea-524f-42cf-854a-65380aca6edc
>2012-12-17 09:02:08,258 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] spmStart polling ended: taskId = c67ed1ea-524f-42cf-854a-65380aca6edc task status = finished
>2012-12-17 09:02:08,278 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] spmStart polling ended, spm status: SPM
>2012-12-17 09:02:08,280 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c67ed1ea-524f-42cf-854a-65380aca6edc), log id: 1c2c3f69
>2012-12-17 09:02:08,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] FINISH, HSMClearTaskVDSCommand, log id: 1c2c3f69
>2012-12-17 09:02:08,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1cf714f2, log id: 45954737
>2012-12-17 09:02:08,304 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [32c2c5a9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:02:08,378 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [55ed84] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:02:08,378 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6c8ac91f
>2012-12-17 09:02:08,395 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [32c2c5a9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:02:08,583 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:02:08,662 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 44cf968a
>2012-12-17 09:02:08,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 36ac3816
>2012-12-17 09:02:08,684 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:09,169 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] FINISH, SpmStopVDSCommand, log id: 36ac3816
>2012-12-17 09:02:09,219 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] FINISH, ResetIrsVDSCommand, log id: 44cf968a
>2012-12-17 09:02:09,230 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 292269af
>2012-12-17 09:02:09,230 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:09,230 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 292269af
>2012-12-17 09:02:09,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:02:09,283 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4728
>2012-12-17 09:02:09,284 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55ed84] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4728, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 47e37e3c
>2012-12-17 09:02:09,305 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55ed84] spmStart polling started: taskId = 17456770-127a-450e-9c90-76948fd1c7ee
>2012-12-17 09:02:10,337 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55ed84] spmStart polling ended: taskId = 17456770-127a-450e-9c90-76948fd1c7ee task status = finished
>2012-12-17 09:02:10,356 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55ed84] spmStart polling ended, spm status: SPM
>2012-12-17 09:02:10,358 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [55ed84] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=17456770-127a-450e-9c90-76948fd1c7ee), log id: 6722c6d6
>2012-12-17 09:02:10,384 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [55ed84] FINISH, HSMClearTaskVDSCommand, log id: 6722c6d6
>2012-12-17 09:02:10,384 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55ed84] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c2c4080, log id: 47e37e3c
>2012-12-17 09:02:10,872 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:02:10,923 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:10,925 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 22f32b19
>2012-12-17 09:02:10,946 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 22f32b19
>2012-12-17 09:02:10,947 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6c8ac91f
>2012-12-17 09:02:10,947 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [55ed84] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:02:10,947 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: a92883c
>2012-12-17 09:02:10,947 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:10,948 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4e36259d
>2012-12-17 09:02:10,966 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4e36259d
>2012-12-17 09:02:10,967 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a92883c
>2012-12-17 09:02:10,967 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [55ed84] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:02:19,306 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [49b46a39] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:02:19,396 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-36) [505260a3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:02:19,452 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 416a8c3b
>2012-12-17 09:02:19,454 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 225c9b93
>2012-12-17 09:02:19,473 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [505260a3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:19,626 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, SpmStopVDSCommand, log id: 225c9b93
>2012-12-17 09:02:19,634 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, ResetIrsVDSCommand, log id: 416a8c3b
>2012-12-17 09:02:19,645 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: c1478a4
>2012-12-17 09:02:19,645 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [505260a3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:19,645 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: c1478a4
>2012-12-17 09:02:20,220 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5abe3acd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:02:29,686 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [67d5cb38] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:02:29,702 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [67d5cb38] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4729
>2012-12-17 09:02:29,703 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4729, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b332b95
>2012-12-17 09:02:29,724 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] spmStart polling started: taskId = 2e5233b6-3088-46a7-bcf1-4222a037bb4a
>2012-12-17 09:02:30,745 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] spmStart polling ended: taskId = 2e5233b6-3088-46a7-bcf1-4222a037bb4a task status = finished
>2012-12-17 09:02:30,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] spmStart polling ended, spm status: SPM
>2012-12-17 09:02:30,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2e5233b6-3088-46a7-bcf1-4222a037bb4a), log id: 56cecef1
>2012-12-17 09:02:30,786 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] FINISH, HSMClearTaskVDSCommand, log id: 56cecef1
>2012-12-17 09:02:30,786 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3672c99e, log id: 6b332b95
>2012-12-17 09:02:31,454 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [67d5cb38] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:02:31,568 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-26) [2ea5eaf2] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:02:31,568 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5073e296
>2012-12-17 09:02:31,600 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [67d5cb38] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:02:31,756 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-28) [57f1ef14] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:02:31,808 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [57f1ef14] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 55c490a6
>2012-12-17 09:02:31,810 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [57f1ef14] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 369f524
>2012-12-17 09:02:31,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [57f1ef14] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:31,972 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [57f1ef14] FINISH, SpmStopVDSCommand, log id: 369f524
>2012-12-17 09:02:31,993 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [57f1ef14] FINISH, ResetIrsVDSCommand, log id: 55c490a6
>2012-12-17 09:02:32,004 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [57f1ef14] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5f3e82ca
>2012-12-17 09:02:32,004 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [57f1ef14] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:32,004 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [57f1ef14] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5f3e82ca
>2012-12-17 09:02:32,067 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:02:32,077 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4730
>2012-12-17 09:02:32,079 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4730, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5bb87508
>2012-12-17 09:02:32,107 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] spmStart polling started: taskId = 58a3130d-3f95-45c5-8a02-5d8440985fd2
>2012-12-17 09:02:33,128 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] spmStart polling ended: taskId = 58a3130d-3f95-45c5-8a02-5d8440985fd2 task status = finished
>2012-12-17 09:02:33,146 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] spmStart polling ended, spm status: SPM
>2012-12-17 09:02:33,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=58a3130d-3f95-45c5-8a02-5d8440985fd2), log id: 648a41ea
>2012-12-17 09:02:33,166 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, HSMClearTaskVDSCommand, log id: 648a41ea
>2012-12-17 09:02:33,166 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@30ed9d37, log id: 5bb87508
>2012-12-17 09:02:33,174 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:02:33,188 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:33,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3a78df43
>2012-12-17 09:02:33,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3a78df43
>2012-12-17 09:02:33,207 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5073e296
>2012-12-17 09:02:33,207 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-26) [2ea5eaf2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:02:33,208 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 481254fd
>2012-12-17 09:02:33,208 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:33,209 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2467b64b
>2012-12-17 09:02:33,227 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2467b64b
>2012-12-17 09:02:33,227 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 481254fd
>2012-12-17 09:02:33,227 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-26) [2ea5eaf2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:02:41,688 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5044c757] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:02:42,113 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [6fcdea84] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:02:42,238 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-44) [6bebc102] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:02:42,304 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2348513c
>2012-12-17 09:02:42,306 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 24ce4457
>2012-12-17 09:02:42,326 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:42,528 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] FINISH, SpmStopVDSCommand, log id: 24ce4457
>2012-12-17 09:02:42,543 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] FINISH, ResetIrsVDSCommand, log id: 2348513c
>2012-12-17 09:02:42,553 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5a3ce1e0
>2012-12-17 09:02:42,554 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [6bebc102] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:42,554 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5a3ce1e0
>2012-12-17 09:02:51,866 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [53d0bc2b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:02:52,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:02:52,613 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4731
>2012-12-17 09:02:52,614 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4731, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6fcf8237
>2012-12-17 09:02:52,636 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] spmStart polling started: taskId = cb044a7b-50a1-4283-9c1d-9fc80c045f9b
>2012-12-17 09:02:53,658 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] spmStart polling ended: taskId = cb044a7b-50a1-4283-9c1d-9fc80c045f9b task status = finished
>2012-12-17 09:02:53,676 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] spmStart polling ended, spm status: SPM
>2012-12-17 09:02:53,677 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cb044a7b-50a1-4283-9c1d-9fc80c045f9b), log id: 2a55a455
>2012-12-17 09:02:53,695 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, HSMClearTaskVDSCommand, log id: 2a55a455
>2012-12-17 09:02:53,695 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5c97de29, log id: 6fcf8237
>2012-12-17 09:02:53,713 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:02:53,742 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [441eec2e] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:02:53,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 686a43e2
>2012-12-17 09:02:53,771 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:02:53,887 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [757df811] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:02:53,967 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [757df811] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 421aa903
>2012-12-17 09:02:53,968 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [757df811] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 314bf78d
>2012-12-17 09:02:53,988 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [757df811] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:54,114 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [757df811] FINISH, SpmStopVDSCommand, log id: 314bf78d
>2012-12-17 09:02:54,145 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [757df811] FINISH, ResetIrsVDSCommand, log id: 421aa903
>2012-12-17 09:02:54,160 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [757df811] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6f67378d
>2012-12-17 09:02:54,160 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [757df811] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:54,161 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [757df811] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6f67378d
>2012-12-17 09:02:54,233 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:02:54,239 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4732
>2012-12-17 09:02:54,240 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4732, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2b837fc7
>2012-12-17 09:02:54,262 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] spmStart polling started: taskId = f28b0566-733f-4f12-9fb8-12340561123a
>2012-12-17 09:02:55,285 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] spmStart polling ended: taskId = f28b0566-733f-4f12-9fb8-12340561123a task status = finished
>2012-12-17 09:02:55,303 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] spmStart polling ended, spm status: SPM
>2012-12-17 09:02:55,304 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f28b0566-733f-4f12-9fb8-12340561123a), log id: 604639a
>2012-12-17 09:02:55,322 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, HSMClearTaskVDSCommand, log id: 604639a
>2012-12-17 09:02:55,322 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@a27168c, log id: 2b837fc7
>2012-12-17 09:02:55,341 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:02:55,481 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:55,486 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 220231cd
>2012-12-17 09:02:55,505 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 220231cd
>2012-12-17 09:02:55,505 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 686a43e2
>2012-12-17 09:02:55,505 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [441eec2e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:02:55,505 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3e139b2c
>2012-12-17 09:02:55,505 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:02:55,507 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 32144f57
>2012-12-17 09:02:55,525 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 32144f57
>2012-12-17 09:02:55,525 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3e139b2c
>2012-12-17 09:02:55,525 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [441eec2e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:03:02,083 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [579f6ed1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:03:04,273 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [786fe4b5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:03:04,420 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [72736c3b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:03:04,509 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [72736c3b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17c8d79a
>2012-12-17 09:03:04,510 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [72736c3b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 752c98b
>2012-12-17 09:03:04,530 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [72736c3b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:03:04,668 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [72736c3b] FINISH, SpmStopVDSCommand, log id: 752c98b
>2012-12-17 09:03:04,685 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [72736c3b] FINISH, ResetIrsVDSCommand, log id: 17c8d79a
>2012-12-17 09:03:04,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [72736c3b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 66c9aeaf
>2012-12-17 09:03:04,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [72736c3b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:03:04,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [72736c3b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 66c9aeaf
>2012-12-17 09:03:12,282 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4256c72b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:03:14,763 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [15510694] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:03:14,789 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [15510694] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4733
>2012-12-17 09:03:14,791 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [15510694] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4733, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7627d942
>2012-12-17 09:03:14,814 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [15510694] spmStart polling started: taskId = 102587cf-f5f1-4973-98c6-81f0be820d82
>2012-12-17 09:03:15,835 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [15510694] spmStart polling ended: taskId = 102587cf-f5f1-4973-98c6-81f0be820d82 task status = finished
>2012-12-17 09:03:15,854 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [15510694] spmStart polling ended, spm status: SPM
>2012-12-17 09:03:15,855 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [15510694] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=102587cf-f5f1-4973-98c6-81f0be820d82), log id: 1d8e2ef9
>2012-12-17 09:03:15,873 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [15510694] FINISH, HSMClearTaskVDSCommand, log id: 1d8e2ef9
>2012-12-17 09:03:15,873 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [15510694] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@61adec97, log id: 7627d942
>2012-12-17 09:03:16,079 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [15510694] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:03:16,145 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [5e38fcf2] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:03:16,145 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 341244a9
>2012-12-17 09:03:16,180 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [15510694] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:03:16,328 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [3808661d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:03:16,384 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [3808661d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 32b7c0eb
>2012-12-17 09:03:16,386 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [3808661d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5e73bf18
>2012-12-17 09:03:16,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [3808661d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:03:16,515 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [3808661d] FINISH, SpmStopVDSCommand, log id: 5e73bf18
>2012-12-17 09:03:16,528 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [3808661d] FINISH, ResetIrsVDSCommand, log id: 32b7c0eb
>2012-12-17 09:03:16,539 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [3808661d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: b2c98da
>2012-12-17 09:03:16,539 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3808661d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:03:16,539 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [3808661d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: b2c98da
>2012-12-17 09:03:16,598 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:03:16,611 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4734
>2012-12-17 09:03:16,612 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4734, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4eac4dc6
>2012-12-17 09:03:16,635 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] spmStart polling started: taskId = 2fd5d38f-ba56-4eec-804b-9d9a31821346
>2012-12-17 09:03:17,657 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] spmStart polling ended: taskId = 2fd5d38f-ba56-4eec-804b-9d9a31821346 task status = finished
>2012-12-17 09:03:17,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] spmStart polling ended, spm status: SPM
>2012-12-17 09:03:17,676 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2fd5d38f-ba56-4eec-804b-9d9a31821346), log id: 22005947
>2012-12-17 09:03:17,693 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, HSMClearTaskVDSCommand, log id: 22005947
>2012-12-17 09:03:17,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@44e555c1, log id: 4eac4dc6
>2012-12-17 09:03:17,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:03:17,725 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:03:17,727 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 21ebd76d
>2012-12-17 09:03:17,745 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 21ebd76d
>2012-12-17 09:03:17,745 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 341244a9
>2012-12-17 09:03:17,745 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [5e38fcf2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:03:17,745 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4768c075
>2012-12-17 09:03:17,745 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:03:17,747 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5c1aba78
>2012-12-17 09:03:17,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5c1aba78
>2012-12-17 09:03:17,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4768c075
>2012-12-17 09:03:17,765 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [5e38fcf2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:03:22,474 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [50b67ada] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:03:26,677 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [107aac36] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:03:26,787 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [7e0f01b0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:03:26,843 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [7e0f01b0] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2cdc1d0e
>2012-12-17 09:03:26,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [7e0f01b0] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3cdb2c49
>2012-12-17 09:03:26,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [7e0f01b0] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:03:27,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [7e0f01b0] FINISH, SpmStopVDSCommand, log id: 3cdb2c49
>2012-12-17 09:03:27,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [7e0f01b0] FINISH, ResetIrsVDSCommand, log id: 2cdc1d0e
>2012-12-17 09:03:27,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [7e0f01b0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 13364e48
>2012-12-17 09:03:27,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7e0f01b0] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:03:27,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [7e0f01b0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 13364e48
>2012-12-17 09:03:32,683 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [108329de] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:03:37,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [490300dd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:03:37,508 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [490300dd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4735
>2012-12-17 09:03:37,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [490300dd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4735, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3c1b5910
>2012-12-17 09:03:37,531 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [490300dd] spmStart polling started: taskId = e9132787-165c-4a55-afb7-a64717521835
>2012-12-17 09:03:38,553 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [490300dd] spmStart polling ended: taskId = e9132787-165c-4a55-afb7-a64717521835 task status = finished
>2012-12-17 09:03:38,571 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [490300dd] spmStart polling ended, spm status: SPM
>2012-12-17 09:03:38,573 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [490300dd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e9132787-165c-4a55-afb7-a64717521835), log id: 78dee0f7
>2012-12-17 09:03:38,591 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [490300dd] FINISH, HSMClearTaskVDSCommand, log id: 78dee0f7
>2012-12-17 09:03:38,592 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [490300dd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7083b61f, log id: 3c1b5910
>2012-12-17 09:03:38,607 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [490300dd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:03:38,676 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:03:38,676 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1ba6a883
>2012-12-17 09:03:38,718 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [490300dd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:03:38,886 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [2c8e27f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:03:38,946 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [2c8e27f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3cf273ed
>2012-12-17 09:03:38,947 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [2c8e27f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4508b00d
>2012-12-17 09:03:38,967 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [2c8e27f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:03:39,084 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [2c8e27f] FINISH, SpmStopVDSCommand, log id: 4508b00d
>2012-12-17 09:03:39,127 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [2c8e27f] FINISH, ResetIrsVDSCommand, log id: 3cf273ed
>2012-12-17 09:03:39,139 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [2c8e27f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 676a54d
>2012-12-17 09:03:39,139 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [2c8e27f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:03:39,139 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [2c8e27f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 676a54d
>2012-12-17 09:03:39,221 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:03:39,284 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4736
>2012-12-17 09:03:39,285 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4736, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6f6e157a
>2012-12-17 09:03:39,308 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling started: taskId = 57270bb2-8d30-4b32-9feb-eb3f4537d5dd
>2012-12-17 09:03:40,331 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended: taskId = 57270bb2-8d30-4b32-9feb-eb3f4537d5dd task status = finished
>2012-12-17 09:03:40,350 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended, spm status: SPM
>2012-12-17 09:03:40,351 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=57270bb2-8d30-4b32-9feb-eb3f4537d5dd), log id: 5ba082a2
>2012-12-17 09:03:40,370 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMClearTaskVDSCommand, log id: 5ba082a2
>2012-12-17 09:03:40,370 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6e088ad, log id: 6f6e157a
>2012-12-17 09:03:40,911 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:03:41,289 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:03:41,291 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2d12b777
>2012-12-17 09:03:41,311 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2d12b777
>2012-12-17 09:03:41,311 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1ba6a883
>2012-12-17 09:03:41,311 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:03:41,311 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 27cafcbf
>2012-12-17 09:03:41,311 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:03:41,312 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4b00a409
>2012-12-17 09:03:41,331 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b00a409
>2012-12-17 09:03:41,331 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 27cafcbf
>2012-12-17 09:03:41,331 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:03:42,881 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:03:49,031 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-14) [3d41845a] Running command: LogoutUserCommand internal: false.
>2012-12-17 09:03:49,259 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [579f6ed1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:03:49,354 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [94e4f4d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:03:49,406 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [94e4f4d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 369b804d
>2012-12-17 09:03:49,407 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [94e4f4d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: e80bed9
>2012-12-17 09:03:49,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [94e4f4d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:03:49,548 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [94e4f4d] FINISH, SpmStopVDSCommand, log id: e80bed9
>2012-12-17 09:03:49,563 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [94e4f4d] FINISH, ResetIrsVDSCommand, log id: 369b804d
>2012-12-17 09:03:49,573 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [94e4f4d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ae17413
>2012-12-17 09:03:49,573 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [94e4f4d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:03:49,573 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [94e4f4d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ae17413
>2012-12-17 09:03:53,072 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4beddd99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:03:59,628 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [30604458] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:03:59,639 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [30604458] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4737
>2012-12-17 09:03:59,641 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [30604458] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4737, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 49bd1acf
>2012-12-17 09:03:59,663 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [30604458] spmStart polling started: taskId = 25bc733a-33b3-4a04-ba77-d5fc3cf4d525
>2012-12-17 09:04:00,686 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [30604458] spmStart polling ended: taskId = 25bc733a-33b3-4a04-ba77-d5fc3cf4d525 task status = finished
>2012-12-17 09:04:00,704 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [30604458] spmStart polling ended, spm status: SPM
>2012-12-17 09:04:00,948 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [30604458] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=25bc733a-33b3-4a04-ba77-d5fc3cf4d525), log id: 268a5401
>2012-12-17 09:04:00,968 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [30604458] FINISH, HSMClearTaskVDSCommand, log id: 268a5401
>2012-12-17 09:04:00,968 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [30604458] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4391216b, log id: 49bd1acf
>2012-12-17 09:04:01,075 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [30604458] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:04:01,204 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:04:01,204 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 663c46a0
>2012-12-17 09:04:01,223 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [30604458] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:04:01,386 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-49) [33cde09c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:04:01,476 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 450cf449
>2012-12-17 09:04:01,478 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3ef98b19
>2012-12-17 09:04:01,497 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:01,606 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SpmStopVDSCommand, log id: 3ef98b19
>2012-12-17 09:04:01,626 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, ResetIrsVDSCommand, log id: 450cf449
>2012-12-17 09:04:01,637 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3591cab8
>2012-12-17 09:04:01,637 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:01,637 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3591cab8
>2012-12-17 09:04:01,680 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:04:01,692 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4738
>2012-12-17 09:04:01,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4738, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 489a2fa5
>2012-12-17 09:04:01,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] spmStart polling started: taskId = 06e27fcc-d6ac-4c25-848e-d3b754444ae5
>2012-12-17 09:04:02,738 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] spmStart polling ended: taskId = 06e27fcc-d6ac-4c25-848e-d3b754444ae5 task status = finished
>2012-12-17 09:04:02,756 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] spmStart polling ended, spm status: SPM
>2012-12-17 09:04:02,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=06e27fcc-d6ac-4c25-848e-d3b754444ae5), log id: 2c6dae83
>2012-12-17 09:04:02,776 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMClearTaskVDSCommand, log id: 2c6dae83
>2012-12-17 09:04:02,776 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@54dc313e, log id: 489a2fa5
>2012-12-17 09:04:02,788 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:04:02,798 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:02,802 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 14392b54
>2012-12-17 09:04:02,821 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 14392b54
>2012-12-17 09:04:02,821 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 663c46a0
>2012-12-17 09:04:02,821 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:04:02,821 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7883d062
>2012-12-17 09:04:02,822 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:02,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1b1d8a50
>2012-12-17 09:04:02,841 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1b1d8a50
>2012-12-17 09:04:02,841 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7883d062
>2012-12-17 09:04:02,841 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:04:03,278 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3808661d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:04:11,723 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:04:11,822 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [13d04fcc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:04:11,888 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6b9e4925
>2012-12-17 09:04:11,890 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 38610a45
>2012-12-17 09:04:11,910 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:12,110 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] FINISH, SpmStopVDSCommand, log id: 38610a45
>2012-12-17 09:04:12,121 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] FINISH, ResetIrsVDSCommand, log id: 6b9e4925
>2012-12-17 09:04:12,133 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5687f090
>2012-12-17 09:04:12,133 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [13d04fcc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:12,133 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5687f090
>2012-12-17 09:04:13,503 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:04:22,177 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:04:22,191 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4739
>2012-12-17 09:04:22,198 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4739, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 156bb43f
>2012-12-17 09:04:22,220 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] spmStart polling started: taskId = 8a9fe74e-4acf-4bdb-87c2-dfc9984e5f75
>2012-12-17 09:04:23,248 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] spmStart polling ended: taskId = 8a9fe74e-4acf-4bdb-87c2-dfc9984e5f75 task status = finished
>2012-12-17 09:04:23,268 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] spmStart polling ended, spm status: SPM
>2012-12-17 09:04:23,269 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8a9fe74e-4acf-4bdb-87c2-dfc9984e5f75), log id: 4bc6e580
>2012-12-17 09:04:23,288 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, HSMClearTaskVDSCommand, log id: 4bc6e580
>2012-12-17 09:04:23,288 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6f055d5d, log id: 156bb43f
>2012-12-17 09:04:23,309 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:04:23,373 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [4fb47a41] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:04:23,373 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1355fada
>2012-12-17 09:04:23,392 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:04:23,507 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [261e40a1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:04:23,554 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7381198a
>2012-12-17 09:04:23,555 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7721e4df
>2012-12-17 09:04:23,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:23,736 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, SpmStopVDSCommand, log id: 7721e4df
>2012-12-17 09:04:23,759 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, ResetIrsVDSCommand, log id: 7381198a
>2012-12-17 09:04:23,771 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 26424ab7
>2012-12-17 09:04:23,771 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:23,771 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 26424ab7
>2012-12-17 09:04:23,790 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:04:23,876 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:04:23,890 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4740
>2012-12-17 09:04:23,891 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4740, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: c40e8fb
>2012-12-17 09:04:23,913 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] spmStart polling started: taskId = 368cb029-b962-47c4-80ab-22fc9d728d50
>2012-12-17 09:04:24,935 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] spmStart polling ended: taskId = 368cb029-b962-47c4-80ab-22fc9d728d50 task status = finished
>2012-12-17 09:04:24,953 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] spmStart polling ended, spm status: SPM
>2012-12-17 09:04:24,955 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=368cb029-b962-47c4-80ab-22fc9d728d50), log id: a66634
>2012-12-17 09:04:24,973 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, HSMClearTaskVDSCommand, log id: a66634
>2012-12-17 09:04:24,973 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2c7fe041, log id: c40e8fb
>2012-12-17 09:04:24,992 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:04:25,009 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:25,010 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: cfadf1c
>2012-12-17 09:04:25,030 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: cfadf1c
>2012-12-17 09:04:25,030 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1355fada
>2012-12-17 09:04:25,030 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [4fb47a41] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:04:25,030 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2a4dadec
>2012-12-17 09:04:25,030 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:25,031 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1f961a5e
>2012-12-17 09:04:25,034 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [46a80193] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:04:25,035 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2d03bf49
>2012-12-17 09:04:25,050 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1f961a5e
>2012-12-17 09:04:25,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2a4dadec
>2012-12-17 09:04:25,050 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [4fb47a41] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:04:25,051 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:25,052 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d62d5fd
>2012-12-17 09:04:25,071 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d62d5fd
>2012-12-17 09:04:25,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2d03bf49
>2012-12-17 09:04:25,071 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [46a80193] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:04:25,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3cf04f1
>2012-12-17 09:04:25,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:25,073 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3e6c9393
>2012-12-17 09:04:25,091 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3e6c9393
>2012-12-17 09:04:25,091 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3cf04f1
>2012-12-17 09:04:25,091 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [46a80193] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:04:33,942 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:04:34,222 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-60) [670d1744] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:04:34,333 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [670d1744] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: dc6613c
>2012-12-17 09:04:34,334 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [670d1744] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: c3530fd
>2012-12-17 09:04:34,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [670d1744] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:34,550 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [670d1744] FINISH, SpmStopVDSCommand, log id: c3530fd
>2012-12-17 09:04:34,588 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [670d1744] FINISH, ResetIrsVDSCommand, log id: dc6613c
>2012-12-17 09:04:34,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [670d1744] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6b8cc53a
>2012-12-17 09:04:34,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [670d1744] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:34,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [670d1744] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6b8cc53a
>2012-12-17 09:04:44,287 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:04:44,668 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5abe3acd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:04:44,682 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5abe3acd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4741
>2012-12-17 09:04:44,683 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4741, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6da82b42
>2012-12-17 09:04:44,705 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] spmStart polling started: taskId = 3cc41031-fe0b-438f-99bf-f31af2183103
>2012-12-17 09:04:45,727 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] spmStart polling ended: taskId = 3cc41031-fe0b-438f-99bf-f31af2183103 task status = finished
>2012-12-17 09:04:45,745 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] spmStart polling ended, spm status: SPM
>2012-12-17 09:04:45,746 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3cc41031-fe0b-438f-99bf-f31af2183103), log id: 94fc038
>2012-12-17 09:04:45,764 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] FINISH, HSMClearTaskVDSCommand, log id: 94fc038
>2012-12-17 09:04:45,764 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@52d4a14e, log id: 6da82b42
>2012-12-17 09:04:46,092 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5abe3acd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:04:46,460 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:04:46,460 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 74540a08
>2012-12-17 09:04:46,505 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5abe3acd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:04:46,684 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [371b8996] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:04:46,794 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [371b8996] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7c79be22
>2012-12-17 09:04:46,796 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [371b8996] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3312ca9e
>2012-12-17 09:04:46,821 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [371b8996] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:47,511 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [371b8996] FINISH, SpmStopVDSCommand, log id: 3312ca9e
>2012-12-17 09:04:47,532 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [371b8996] FINISH, ResetIrsVDSCommand, log id: 7c79be22
>2012-12-17 09:04:47,543 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [371b8996] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d8a840f
>2012-12-17 09:04:47,543 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [371b8996] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:47,543 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [371b8996] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d8a840f
>2012-12-17 09:04:47,608 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:04:47,646 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4742
>2012-12-17 09:04:47,647 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4742, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 13e5687a
>2012-12-17 09:04:47,669 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling started: taskId = c060be6f-7791-4aae-b5d9-8ec4157f7f52
>2012-12-17 09:04:48,690 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended: taskId = c060be6f-7791-4aae-b5d9-8ec4157f7f52 task status = finished
>2012-12-17 09:04:48,709 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended, spm status: SPM
>2012-12-17 09:04:48,710 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c060be6f-7791-4aae-b5d9-8ec4157f7f52), log id: 598b4d83
>2012-12-17 09:04:48,729 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMClearTaskVDSCommand, log id: 598b4d83
>2012-12-17 09:04:48,729 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@488e0162, log id: 13e5687a
>2012-12-17 09:04:48,774 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:04:48,787 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:48,791 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 611b0ea8
>2012-12-17 09:04:48,811 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 611b0ea8
>2012-12-17 09:04:48,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 74540a08
>2012-12-17 09:04:48,811 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:04:48,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5900c3f0
>2012-12-17 09:04:48,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:48,812 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1a906286
>2012-12-17 09:04:48,831 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1a906286
>2012-12-17 09:04:48,831 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5900c3f0
>2012-12-17 09:04:48,831 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:04:54,534 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [356a6c1f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:04:57,650 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [50b67ada] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:04:58,012 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-29) [57cdd675] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:04:58,143 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7ef78ee
>2012-12-17 09:04:58,145 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7f61f52d
>2012-12-17 09:04:58,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:58,324 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, SpmStopVDSCommand, log id: 7f61f52d
>2012-12-17 09:04:58,342 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, ResetIrsVDSCommand, log id: 7ef78ee
>2012-12-17 09:04:58,353 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6802d1a9
>2012-12-17 09:04:58,353 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:04:58,353 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6802d1a9
>2012-12-17 09:05:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [37630ba1] Checking autorecoverable hosts
>2012-12-17 09:05:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [37630ba1] Autorecovering 0 hosts
>2012-12-17 09:05:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [37630ba1] Checking autorecoverable hosts done
>2012-12-17 09:05:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [37630ba1] Autorecovering storage domains is disabled, skipping
>2012-12-17 09:05:04,731 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:05:08,450 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [108329de] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:05:08,463 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [108329de] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4743
>2012-12-17 09:05:08,464 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [108329de] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4743, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 428e22e9
>2012-12-17 09:05:08,488 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [108329de] spmStart polling started: taskId = 6b114bce-dc6d-4c5e-ba29-107e9efedff0
>2012-12-17 09:05:09,511 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [108329de] spmStart polling ended: taskId = 6b114bce-dc6d-4c5e-ba29-107e9efedff0 task status = finished
>2012-12-17 09:05:09,530 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [108329de] spmStart polling ended, spm status: SPM
>2012-12-17 09:05:09,531 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [108329de] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6b114bce-dc6d-4c5e-ba29-107e9efedff0), log id: 6ddcee7b
>2012-12-17 09:05:09,550 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [108329de] FINISH, HSMClearTaskVDSCommand, log id: 6ddcee7b
>2012-12-17 09:05:09,550 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [108329de] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@43ad0596, log id: 428e22e9
>2012-12-17 09:05:09,560 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [108329de] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:05:09,599 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [261e40a1] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:05:09,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6532d26c
>2012-12-17 09:05:09,618 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [108329de] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:05:09,735 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [73cd7795] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:05:09,788 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 41f6c478
>2012-12-17 09:05:09,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7559310a
>2012-12-17 09:05:09,810 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:09,956 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] FINISH, SpmStopVDSCommand, log id: 7559310a
>2012-12-17 09:05:09,971 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] FINISH, ResetIrsVDSCommand, log id: 41f6c478
>2012-12-17 09:05:09,982 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 205ca197
>2012-12-17 09:05:09,982 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [73cd7795] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:09,982 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 205ca197
>2012-12-17 09:05:10,020 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:05:10,030 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4744
>2012-12-17 09:05:10,031 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4744, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7dac2afd
>2012-12-17 09:05:10,054 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] spmStart polling started: taskId = 9dbd1ce7-6978-4cfd-82a4-aabf9c32447d
>2012-12-17 09:05:11,075 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] spmStart polling ended: taskId = 9dbd1ce7-6978-4cfd-82a4-aabf9c32447d task status = finished
>2012-12-17 09:05:11,094 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] spmStart polling ended, spm status: SPM
>2012-12-17 09:05:11,095 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9dbd1ce7-6978-4cfd-82a4-aabf9c32447d), log id: 74fd0b4d
>2012-12-17 09:05:11,114 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, HSMClearTaskVDSCommand, log id: 74fd0b4d
>2012-12-17 09:05:11,114 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@10f1283c, log id: 7dac2afd
>2012-12-17 09:05:11,174 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:05:11,201 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:11,205 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 71261222
>2012-12-17 09:05:11,225 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 71261222
>2012-12-17 09:05:11,225 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6532d26c
>2012-12-17 09:05:11,225 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [261e40a1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:05:11,225 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7a70b036
>2012-12-17 09:05:11,225 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:11,227 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 422adb4b
>2012-12-17 09:05:11,246 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 422adb4b
>2012-12-17 09:05:11,246 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7a70b036
>2012-12-17 09:05:11,246 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [261e40a1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:05:14,932 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:05:20,082 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [48c4b2a7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:05:20,297 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [3b88f2b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:05:20,358 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [3b88f2b5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5733c68b
>2012-12-17 09:05:20,359 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [3b88f2b5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 25df7980
>2012-12-17 09:05:20,379 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [3b88f2b5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:20,532 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [3b88f2b5] FINISH, SpmStopVDSCommand, log id: 25df7980
>2012-12-17 09:05:21,126 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [3b88f2b5] FINISH, ResetIrsVDSCommand, log id: 5733c68b
>2012-12-17 09:05:21,137 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [3b88f2b5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2d601072
>2012-12-17 09:05:21,138 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [3b88f2b5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:21,138 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [3b88f2b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2d601072
>2012-12-17 09:05:25,169 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:05:31,223 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:05:31,234 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4745
>2012-12-17 09:05:31,235 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4745, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 679c6a22
>2012-12-17 09:05:31,258 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] spmStart polling started: taskId = 274cc9cf-51b7-4394-bfd6-d07379fdbfa7
>2012-12-17 09:05:32,280 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] spmStart polling ended: taskId = 274cc9cf-51b7-4394-bfd6-d07379fdbfa7 task status = finished
>2012-12-17 09:05:32,302 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] spmStart polling ended, spm status: SPM
>2012-12-17 09:05:32,304 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=274cc9cf-51b7-4394-bfd6-d07379fdbfa7), log id: 79ef136c
>2012-12-17 09:05:32,323 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, HSMClearTaskVDSCommand, log id: 79ef136c
>2012-12-17 09:05:32,323 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3ee455fe, log id: 679c6a22
>2012-12-17 09:05:32,346 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:05:32,424 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:05:32,424 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e93a643
>2012-12-17 09:05:32,440 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:05:32,623 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-20) [537845d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:05:32,728 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [537845d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 377cc50a
>2012-12-17 09:05:32,730 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [537845d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5ba6b79a
>2012-12-17 09:05:32,750 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [537845d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:33,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [537845d] FINISH, SpmStopVDSCommand, log id: 5ba6b79a
>2012-12-17 09:05:33,218 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [537845d] FINISH, ResetIrsVDSCommand, log id: 377cc50a
>2012-12-17 09:05:33,228 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [537845d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6693a3f2
>2012-12-17 09:05:33,228 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [537845d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:33,229 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [537845d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6693a3f2
>2012-12-17 09:05:33,304 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:05:33,317 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4746
>2012-12-17 09:05:33,319 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4746, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2530890b
>2012-12-17 09:05:33,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling started: taskId = 1e301a73-da6e-4d79-aa12-4767f7a52065
>2012-12-17 09:05:34,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling ended: taskId = 1e301a73-da6e-4d79-aa12-4767f7a52065 task status = finished
>2012-12-17 09:05:34,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling ended, spm status: SPM
>2012-12-17 09:05:34,382 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1e301a73-da6e-4d79-aa12-4767f7a52065), log id: 129d346b
>2012-12-17 09:05:34,400 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMClearTaskVDSCommand, log id: 129d346b
>2012-12-17 09:05:34,400 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@edc6659, log id: 2530890b
>2012-12-17 09:05:34,420 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:05:34,445 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:34,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 50f6fd6d
>2012-12-17 09:05:34,468 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 50f6fd6d
>2012-12-17 09:05:34,468 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e93a643
>2012-12-17 09:05:34,468 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:05:34,469 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 77828a2
>2012-12-17 09:05:34,469 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:34,470 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 55e861d9
>2012-12-17 09:05:34,488 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 55e861d9
>2012-12-17 09:05:34,488 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 77828a2
>2012-12-17 09:05:34,488 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:05:35,379 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61acfe5d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:05:43,352 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [15ef2937] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:05:43,533 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [1d6be49e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:05:43,651 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 58e597d6
>2012-12-17 09:05:43,652 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 125d71c
>2012-12-17 09:05:43,672 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:43,828 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, SpmStopVDSCommand, log id: 125d71c
>2012-12-17 09:05:43,844 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, ResetIrsVDSCommand, log id: 58e597d6
>2012-12-17 09:05:43,855 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 27e22339
>2012-12-17 09:05:43,855 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1d6be49e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:43,855 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 27e22339
>2012-12-17 09:05:46,195 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:05:53,935 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [6bebc102] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:05:53,943 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [6bebc102] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4747
>2012-12-17 09:05:53,945 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4747, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 23f0c438
>2012-12-17 09:05:53,968 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] spmStart polling started: taskId = e6a86dc0-f041-41c0-8cbe-b334f098e16c
>2012-12-17 09:05:54,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] spmStart polling ended: taskId = e6a86dc0-f041-41c0-8cbe-b334f098e16c task status = finished
>2012-12-17 09:05:55,009 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] spmStart polling ended, spm status: SPM
>2012-12-17 09:05:55,010 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e6a86dc0-f041-41c0-8cbe-b334f098e16c), log id: a4151a9
>2012-12-17 09:05:55,029 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] FINISH, HSMClearTaskVDSCommand, log id: a4151a9
>2012-12-17 09:05:55,030 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@46583381, log id: 23f0c438
>2012-12-17 09:05:55,043 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [6bebc102] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:05:55,079 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-47) [1925ecb1] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:05:55,079 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 69841251
>2012-12-17 09:05:55,101 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [6bebc102] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:05:55,188 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:05:55,241 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6e48ddae
>2012-12-17 09:05:55,242 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7ed3459
>2012-12-17 09:05:55,264 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:55,416 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SpmStopVDSCommand, log id: 7ed3459
>2012-12-17 09:05:56,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, ResetIrsVDSCommand, log id: 6e48ddae
>2012-12-17 09:05:56,282 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 517c2211
>2012-12-17 09:05:56,282 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:56,282 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 517c2211
>2012-12-17 09:05:56,415 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:05:56,428 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:05:56,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4748
>2012-12-17 09:05:56,438 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4748, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3cd6b18a
>2012-12-17 09:05:56,461 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] spmStart polling started: taskId = 1998cb60-f0b3-480a-86b7-8368342962ee
>2012-12-17 09:05:57,488 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] spmStart polling ended: taskId = 1998cb60-f0b3-480a-86b7-8368342962ee task status = finished
>2012-12-17 09:05:57,508 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] spmStart polling ended, spm status: SPM
>2012-12-17 09:05:57,510 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1998cb60-f0b3-480a-86b7-8368342962ee), log id: 18ab78ee
>2012-12-17 09:05:57,533 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, HSMClearTaskVDSCommand, log id: 18ab78ee
>2012-12-17 09:05:57,533 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2debfa98, log id: 3cd6b18a
>2012-12-17 09:05:57,554 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:05:57,593 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:57,598 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5ab1db3b
>2012-12-17 09:05:57,617 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5ab1db3b
>2012-12-17 09:05:57,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 69841251
>2012-12-17 09:05:57,617 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-47) [1925ecb1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:05:57,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 52b02d01
>2012-12-17 09:05:57,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:57,619 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5c16fa37
>2012-12-17 09:05:57,637 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5c16fa37
>2012-12-17 09:05:57,637 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 52b02d01
>2012-12-17 09:05:57,637 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-47) [1925ecb1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:05:57,660 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [7781d588] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:05:57,660 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: c6903e0
>2012-12-17 09:05:57,660 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:57,662 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 394f69fb
>2012-12-17 09:05:57,680 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 394f69fb
>2012-12-17 09:05:57,680 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: c6903e0
>2012-12-17 09:05:57,680 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [7781d588] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:05:57,680 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: cb6fb22
>2012-12-17 09:05:57,681 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:05:57,682 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5e63dbce
>2012-12-17 09:05:57,700 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5e63dbce
>2012-12-17 09:05:57,700 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: cb6fb22
>2012-12-17 09:05:57,700 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [7781d588] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:06:06,517 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [33e6033] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:06:06,860 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [29a7f0f3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:06:07,051 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5a10484a
>2012-12-17 09:06:07,053 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 17bbe054
>2012-12-17 09:06:07,074 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:06:07,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] FINISH, SpmStopVDSCommand, log id: 17bbe054
>2012-12-17 09:06:07,243 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] FINISH, ResetIrsVDSCommand, log id: 5a10484a
>2012-12-17 09:06:07,254 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 35e22bae
>2012-12-17 09:06:07,254 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [29a7f0f3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:06:07,254 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 35e22bae
>2012-12-17 09:06:16,916 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4beddd99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:06:17,319 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:06:17,333 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4749
>2012-12-17 09:06:17,335 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4749, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7f940983
>2012-12-17 09:06:17,357 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling started: taskId = 2e94a711-bbc2-4413-a701-b62972a1755b
>2012-12-17 09:06:18,383 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended: taskId = 2e94a711-bbc2-4413-a701-b62972a1755b task status = finished
>2012-12-17 09:06:18,401 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended, spm status: SPM
>2012-12-17 09:06:18,403 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2e94a711-bbc2-4413-a701-b62972a1755b), log id: 3ffc83ea
>2012-12-17 09:06:18,421 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMClearTaskVDSCommand, log id: 3ffc83ea
>2012-12-17 09:06:18,421 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5b389877, log id: 7f940983
>2012-12-17 09:06:18,448 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:06:18,529 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [b3b23b2] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:06:18,529 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e72cfdf
>2012-12-17 09:06:18,548 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:06:18,791 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-43) [1b349a52] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:06:18,888 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 39630de2
>2012-12-17 09:06:18,891 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 14d4425f
>2012-12-17 09:06:18,912 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:06:19,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] FINISH, SpmStopVDSCommand, log id: 14d4425f
>2012-12-17 09:06:19,048 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] FINISH, ResetIrsVDSCommand, log id: 39630de2
>2012-12-17 09:06:19,081 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2cad9b84
>2012-12-17 09:06:19,081 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:06:19,081 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2cad9b84
>2012-12-17 09:06:19,184 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:06:19,192 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4750
>2012-12-17 09:06:19,194 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4750, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 639aa8b8
>2012-12-17 09:06:19,217 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] spmStart polling started: taskId = 6dc0c6ec-0cd3-410d-b049-9f99fc365a49
>2012-12-17 09:06:20,242 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] spmStart polling ended: taskId = 6dc0c6ec-0cd3-410d-b049-9f99fc365a49 task status = finished
>2012-12-17 09:06:20,260 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] spmStart polling ended, spm status: SPM
>2012-12-17 09:06:20,262 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6dc0c6ec-0cd3-410d-b049-9f99fc365a49), log id: 260db526
>2012-12-17 09:06:20,280 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, HSMClearTaskVDSCommand, log id: 260db526
>2012-12-17 09:06:20,280 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4defe54e, log id: 639aa8b8
>2012-12-17 09:06:20,293 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:06:20,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:06:20,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 10563007
>2012-12-17 09:06:20,352 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 10563007
>2012-12-17 09:06:20,352 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e72cfdf
>2012-12-17 09:06:20,352 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [b3b23b2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:06:20,353 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: a6b23f2
>2012-12-17 09:06:20,353 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:06:20,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 324427c1
>2012-12-17 09:06:20,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 324427c1
>2012-12-17 09:06:20,373 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a6b23f2
>2012-12-17 09:06:20,373 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [b3b23b2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:06:27,169 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:06:29,227 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [57f1ef14] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:06:29,353 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-28) [2e31d140] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:06:29,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 21d27e46
>2012-12-17 09:06:29,548 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 51061dfc
>2012-12-17 09:06:29,567 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:06:29,725 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SpmStopVDSCommand, log id: 51061dfc
>2012-12-17 09:06:29,757 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, ResetIrsVDSCommand, log id: 21d27e46
>2012-12-17 09:06:29,769 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5bb04690
>2012-12-17 09:06:29,769 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:06:29,769 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5bb04690
>2012-12-17 09:06:37,435 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:06:39,838 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [41dfde4b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:06:39,874 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [41dfde4b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4751
>2012-12-17 09:06:39,875 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4751, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2fad926e
>2012-12-17 09:06:39,898 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] spmStart polling started: taskId = 537abddb-57ca-4efb-9163-4f70498ce443
>2012-12-17 09:06:40,921 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] spmStart polling ended: taskId = 537abddb-57ca-4efb-9163-4f70498ce443 task status = finished
>2012-12-17 09:06:40,940 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] spmStart polling ended, spm status: SPM
>2012-12-17 09:06:40,941 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=537abddb-57ca-4efb-9163-4f70498ce443), log id: 662e6f05
>2012-12-17 09:06:40,960 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] FINISH, HSMClearTaskVDSCommand, log id: 662e6f05
>2012-12-17 09:06:40,960 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@727ae5ab, log id: 2fad926e
>2012-12-17 09:06:41,450 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [41dfde4b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:06:41,706 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-34) [5dcd105e] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:06:41,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4a32c779
>2012-12-17 09:06:41,721 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [41dfde4b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:06:41,916 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [1d6e3277] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:06:41,972 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [1d6e3277] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 645d055f
>2012-12-17 09:06:41,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [1d6e3277] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3972d14c
>2012-12-17 09:06:41,993 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [1d6e3277] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:06:42,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [1d6e3277] FINISH, SpmStopVDSCommand, log id: 3972d14c
>2012-12-17 09:06:42,181 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [1d6e3277] FINISH, ResetIrsVDSCommand, log id: 645d055f
>2012-12-17 09:06:42,191 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [1d6e3277] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 120ba2a7
>2012-12-17 09:06:42,191 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [1d6e3277] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:06:42,192 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [1d6e3277] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 120ba2a7
>2012-12-17 09:06:42,238 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [5dcd105e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:06:42,252 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [5dcd105e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4752
>2012-12-17 09:06:42,253 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4752, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44efeb30
>2012-12-17 09:06:42,275 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] spmStart polling started: taskId = 4b9dc5ec-8793-4a53-b631-19bb1e2e25f4
>2012-12-17 09:06:43,297 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] spmStart polling ended: taskId = 4b9dc5ec-8793-4a53-b631-19bb1e2e25f4 task status = finished
>2012-12-17 09:06:43,316 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] spmStart polling ended, spm status: SPM
>2012-12-17 09:06:43,317 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4b9dc5ec-8793-4a53-b631-19bb1e2e25f4), log id: 1345c758
>2012-12-17 09:06:43,335 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, HSMClearTaskVDSCommand, log id: 1345c758
>2012-12-17 09:06:43,336 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4a226c4a, log id: 44efeb30
>2012-12-17 09:06:43,367 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [5dcd105e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:06:43,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:06:43,392 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 48d5d5e6
>2012-12-17 09:06:43,414 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 48d5d5e6
>2012-12-17 09:06:43,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4a32c779
>2012-12-17 09:06:43,414 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-34) [5dcd105e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:06:43,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4ece1465
>2012-12-17 09:06:43,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:06:43,416 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 14df502b
>2012-12-17 09:06:43,438 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 14df502b
>2012-12-17 09:06:43,438 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4ece1465
>2012-12-17 09:06:43,438 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-34) [5dcd105e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:06:47,629 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:06:52,283 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:06:52,385 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-3) [5d11909d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:06:52,423 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 62f0bdbf
>2012-12-17 09:06:52,425 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 39ccd91f
>2012-12-17 09:06:52,445 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:06:52,609 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] FINISH, SpmStopVDSCommand, log id: 39ccd91f
>2012-12-17 09:06:52,644 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] FINISH, ResetIrsVDSCommand, log id: 62f0bdbf
>2012-12-17 09:06:52,655 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 75de7371
>2012-12-17 09:06:52,655 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [5d11909d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:06:52,655 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 75de7371
>2012-12-17 09:06:57,827 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [436410e6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:07:02,741 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:07:02,755 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4753
>2012-12-17 09:07:02,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4753, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3bd00e2d
>2012-12-17 09:07:02,780 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] spmStart polling started: taskId = 70b61ba2-bda1-490d-8228-74eb5da0d29c
>2012-12-17 09:07:03,802 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] spmStart polling ended: taskId = 70b61ba2-bda1-490d-8228-74eb5da0d29c task status = finished
>2012-12-17 09:07:03,820 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] spmStart polling ended, spm status: SPM
>2012-12-17 09:07:03,821 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=70b61ba2-bda1-490d-8228-74eb5da0d29c), log id: 44ba4750
>2012-12-17 09:07:03,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, HSMClearTaskVDSCommand, log id: 44ba4750
>2012-12-17 09:07:03,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@219ff870, log id: 3bd00e2d
>2012-12-17 09:07:03,873 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:07:03,916 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [505260a3] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:07:03,916 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5eba94e1
>2012-12-17 09:07:03,956 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:07:04,069 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [455c512f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:07:04,170 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [455c512f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 37f9d7a
>2012-12-17 09:07:04,171 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [455c512f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 716d05b8
>2012-12-17 09:07:04,191 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [455c512f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:04,350 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [455c512f] FINISH, SpmStopVDSCommand, log id: 716d05b8
>2012-12-17 09:07:04,365 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [455c512f] FINISH, ResetIrsVDSCommand, log id: 37f9d7a
>2012-12-17 09:07:04,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [455c512f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1f787bd9
>2012-12-17 09:07:04,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [455c512f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:04,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [455c512f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1f787bd9
>2012-12-17 09:07:04,476 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [505260a3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:07:04,516 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [505260a3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4754
>2012-12-17 09:07:04,517 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4754, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 28bd51cd
>2012-12-17 09:07:04,541 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [505260a3] spmStart polling started: taskId = ec9ad9d5-ed19-4b93-bfd4-b31691786cde
>2012-12-17 09:07:05,564 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [505260a3] spmStart polling ended: taskId = ec9ad9d5-ed19-4b93-bfd4-b31691786cde task status = finished
>2012-12-17 09:07:05,582 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [505260a3] spmStart polling ended, spm status: SPM
>2012-12-17 09:07:05,583 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ec9ad9d5-ed19-4b93-bfd4-b31691786cde), log id: 48ff5700
>2012-12-17 09:07:05,602 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, HSMClearTaskVDSCommand, log id: 48ff5700
>2012-12-17 09:07:05,602 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@73a8e670, log id: 28bd51cd
>2012-12-17 09:07:06,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [505260a3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:07:06,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:06,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 52ef296
>2012-12-17 09:07:06,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 52ef296
>2012-12-17 09:07:06,449 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5eba94e1
>2012-12-17 09:07:06,449 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [505260a3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:07:06,449 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4d385882
>2012-12-17 09:07:06,449 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:06,451 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 32cec836
>2012-12-17 09:07:06,470 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 32cec836
>2012-12-17 09:07:06,470 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4d385882
>2012-12-17 09:07:06,470 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [505260a3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:07:08,138 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:07:14,551 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3808661d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:07:14,701 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [3aea96b8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:07:14,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 52bdee72
>2012-12-17 09:07:14,767 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 239578d
>2012-12-17 09:07:14,787 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:14,943 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, SpmStopVDSCommand, log id: 239578d
>2012-12-17 09:07:14,977 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, ResetIrsVDSCommand, log id: 52bdee72
>2012-12-17 09:07:14,988 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76abf691
>2012-12-17 09:07:14,989 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:14,989 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76abf691
>2012-12-17 09:07:18,397 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:07:25,063 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:07:25,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4755
>2012-12-17 09:07:25,096 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4755, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5a75936
>2012-12-17 09:07:25,117 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling started: taskId = 423a7f09-5bd2-4134-98bd-17669110ec38
>2012-12-17 09:07:26,140 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling ended: taskId = 423a7f09-5bd2-4134-98bd-17669110ec38 task status = finished
>2012-12-17 09:07:26,161 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling ended, spm status: SPM
>2012-12-17 09:07:26,163 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=423a7f09-5bd2-4134-98bd-17669110ec38), log id: 43153e71
>2012-12-17 09:07:26,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, HSMClearTaskVDSCommand, log id: 43153e71
>2012-12-17 09:07:26,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@27b68f1f, log id: 5a75936
>2012-12-17 09:07:26,210 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:07:26,268 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [39739d98] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:07:26,268 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 618f5025
>2012-12-17 09:07:26,280 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:07:26,528 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [af8a779] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:07:26,733 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [af8a779] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 769de3fe
>2012-12-17 09:07:26,735 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [af8a779] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 596d4e74
>2012-12-17 09:07:26,755 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [af8a779] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:27,028 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [af8a779] FINISH, SpmStopVDSCommand, log id: 596d4e74
>2012-12-17 09:07:27,060 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [af8a779] FINISH, ResetIrsVDSCommand, log id: 769de3fe
>2012-12-17 09:07:27,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [af8a779] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7c1316f8
>2012-12-17 09:07:27,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [af8a779] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:27,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [af8a779] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7c1316f8
>2012-12-17 09:07:27,142 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:07:27,161 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4756
>2012-12-17 09:07:27,162 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4756, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7d62b2b7
>2012-12-17 09:07:27,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] spmStart polling started: taskId = 18dfe81b-0b38-45d0-9463-2705024911f6
>2012-12-17 09:07:28,212 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] spmStart polling ended: taskId = 18dfe81b-0b38-45d0-9463-2705024911f6 task status = finished
>2012-12-17 09:07:28,230 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] spmStart polling ended, spm status: SPM
>2012-12-17 09:07:28,231 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=18dfe81b-0b38-45d0-9463-2705024911f6), log id: 7f957c50
>2012-12-17 09:07:28,250 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, HSMClearTaskVDSCommand, log id: 7f957c50
>2012-12-17 09:07:28,250 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2f14a9a0, log id: 7d62b2b7
>2012-12-17 09:07:28,278 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:07:28,332 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:28,343 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 65e87494
>2012-12-17 09:07:28,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 65e87494
>2012-12-17 09:07:28,362 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 618f5025
>2012-12-17 09:07:28,362 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [39739d98] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:07:28,362 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3dd4d1
>2012-12-17 09:07:28,362 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:28,364 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 307bc725
>2012-12-17 09:07:28,383 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 307bc725
>2012-12-17 09:07:28,383 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3dd4d1
>2012-12-17 09:07:28,383 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [39739d98] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:07:28,609 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [1d6e3277] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:07:37,194 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [2c8e27f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:07:37,300 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [6573753] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:07:37,359 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [6573753] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5004452c
>2012-12-17 09:07:37,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [6573753] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26caba79
>2012-12-17 09:07:37,380 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [6573753] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:38,205 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [6573753] FINISH, SpmStopVDSCommand, log id: 26caba79
>2012-12-17 09:07:38,219 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [6573753] FINISH, ResetIrsVDSCommand, log id: 5004452c
>2012-12-17 09:07:38,231 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [6573753] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5053d0
>2012-12-17 09:07:38,231 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [6573753] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:38,231 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [6573753] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5053d0
>2012-12-17 09:07:38,865 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:07:48,280 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:07:48,293 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4757
>2012-12-17 09:07:48,294 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4757, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5ca4e3d3
>2012-12-17 09:07:48,317 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] spmStart polling started: taskId = cca469a2-f66e-4170-bdc1-7ef45e6634f4
>2012-12-17 09:07:49,338 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] spmStart polling ended: taskId = cca469a2-f66e-4170-bdc1-7ef45e6634f4 task status = finished
>2012-12-17 09:07:49,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] spmStart polling ended, spm status: SPM
>2012-12-17 09:07:49,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cca469a2-f66e-4170-bdc1-7ef45e6634f4), log id: 55386c30
>2012-12-17 09:07:49,380 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, HSMClearTaskVDSCommand, log id: 55386c30
>2012-12-17 09:07:49,380 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6b46ef05, log id: 5ca4e3d3
>2012-12-17 09:07:49,396 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:07:49,440 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [71b7bdef] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:07:49,440 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 338767a6
>2012-12-17 09:07:49,480 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:07:49,690 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-4) [263a433b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:07:49,746 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1164ba76
>2012-12-17 09:07:49,747 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 280d1f6d
>2012-12-17 09:07:49,767 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [263a433b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:49,898 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, SpmStopVDSCommand, log id: 280d1f6d
>2012-12-17 09:07:49,926 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, ResetIrsVDSCommand, log id: 1164ba76
>2012-12-17 09:07:49,938 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 71a5d43b
>2012-12-17 09:07:49,938 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:49,938 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 71a5d43b
>2012-12-17 09:07:49,986 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:07:49,999 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4758
>2012-12-17 09:07:50,001 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4758, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 53c1bc43
>2012-12-17 09:07:50,024 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] spmStart polling started: taskId = efa98d38-80d1-4999-821b-ac7da9b7ed51
>2012-12-17 09:07:51,047 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] spmStart polling ended: taskId = efa98d38-80d1-4999-821b-ac7da9b7ed51 task status = finished
>2012-12-17 09:07:51,081 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] spmStart polling ended, spm status: SPM
>2012-12-17 09:07:51,083 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=efa98d38-80d1-4999-821b-ac7da9b7ed51), log id: 63c3e81d
>2012-12-17 09:07:51,100 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, HSMClearTaskVDSCommand, log id: 63c3e81d
>2012-12-17 09:07:51,101 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@52bbfb16, log id: 53c1bc43
>2012-12-17 09:07:51,165 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:07:51,211 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:51,213 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3ca38fa7
>2012-12-17 09:07:51,232 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3ca38fa7
>2012-12-17 09:07:51,232 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 338767a6
>2012-12-17 09:07:51,232 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [71b7bdef] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:07:51,232 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 321a018c
>2012-12-17 09:07:51,232 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:07:51,233 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2de2cc91
>2012-12-17 09:07:51,252 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2de2cc91
>2012-12-17 09:07:51,252 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 321a018c
>2012-12-17 09:07:51,252 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [71b7bdef] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:07:59,257 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:08:00,035 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:08:00,255 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [24cba1b2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:08:00,327 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4197239
>2012-12-17 09:08:00,328 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 49618fb7
>2012-12-17 09:08:00,348 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:00,532 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] FINISH, SpmStopVDSCommand, log id: 49618fb7
>2012-12-17 09:08:00,866 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] FINISH, ResetIrsVDSCommand, log id: 4197239
>2012-12-17 09:08:00,877 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2ea29a16
>2012-12-17 09:08:00,877 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [24cba1b2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:00,877 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2ea29a16
>2012-12-17 09:08:09,538 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:08:11,123 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [13d04fcc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:08:11,430 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [13d04fcc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4759
>2012-12-17 09:08:11,432 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4759, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: e7f11d
>2012-12-17 09:08:11,455 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] spmStart polling started: taskId = 6791b8d5-ee11-41cb-97ce-b1e603db1259
>2012-12-17 09:08:12,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] spmStart polling ended: taskId = 6791b8d5-ee11-41cb-97ce-b1e603db1259 task status = finished
>2012-12-17 09:08:12,495 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] spmStart polling ended, spm status: SPM
>2012-12-17 09:08:12,497 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6791b8d5-ee11-41cb-97ce-b1e603db1259), log id: cab48dc
>2012-12-17 09:08:12,515 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] FINISH, HSMClearTaskVDSCommand, log id: cab48dc
>2012-12-17 09:08:12,515 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@413c6a03, log id: e7f11d
>2012-12-17 09:08:12,527 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [13d04fcc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:08:12,558 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-41) [5044c757] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:08:12,558 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 59462d2d
>2012-12-17 09:08:12,585 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [13d04fcc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:08:12,699 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [30572eed] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:08:12,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [30572eed] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 31bc4167
>2012-12-17 09:08:12,746 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [30572eed] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 212a69c
>2012-12-17 09:08:12,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [30572eed] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:12,906 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [30572eed] FINISH, SpmStopVDSCommand, log id: 212a69c
>2012-12-17 09:08:12,920 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [30572eed] FINISH, ResetIrsVDSCommand, log id: 31bc4167
>2012-12-17 09:08:12,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [30572eed] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4033f314
>2012-12-17 09:08:12,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [30572eed] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:12,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [30572eed] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4033f314
>2012-12-17 09:08:12,986 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5044c757] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:08:12,998 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5044c757] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4760
>2012-12-17 09:08:12,999 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4760, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67c97c39
>2012-12-17 09:08:13,021 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [5044c757] spmStart polling started: taskId = 64c320dd-1224-4d2d-8244-1d965f1f7cf7
>2012-12-17 09:08:14,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [5044c757] spmStart polling ended: taskId = 64c320dd-1224-4d2d-8244-1d965f1f7cf7 task status = finished
>2012-12-17 09:08:14,060 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [5044c757] spmStart polling ended, spm status: SPM
>2012-12-17 09:08:14,062 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=64c320dd-1224-4d2d-8244-1d965f1f7cf7), log id: 2e4d1123
>2012-12-17 09:08:14,081 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, HSMClearTaskVDSCommand, log id: 2e4d1123
>2012-12-17 09:08:14,081 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@45a02894, log id: 67c97c39
>2012-12-17 09:08:14,101 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5044c757] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:08:14,128 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:14,132 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 341601ea
>2012-12-17 09:08:14,152 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 341601ea
>2012-12-17 09:08:14,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 59462d2d
>2012-12-17 09:08:14,152 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-41) [5044c757] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:08:14,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 459cbe4e
>2012-12-17 09:08:14,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:14,154 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 69951ce5
>2012-12-17 09:08:14,176 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 69951ce5
>2012-12-17 09:08:14,176 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 459cbe4e
>2012-12-17 09:08:14,176 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-41) [5044c757] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:08:19,791 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:08:23,029 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [53d0bc2b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:08:23,163 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [68b3db6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:08:23,235 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 54367191
>2012-12-17 09:08:23,237 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3f149e9d
>2012-12-17 09:08:23,256 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:23,437 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, SpmStopVDSCommand, log id: 3f149e9d
>2012-12-17 09:08:23,488 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, ResetIrsVDSCommand, log id: 54367191
>2012-12-17 09:08:23,498 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5889c3b6
>2012-12-17 09:08:23,499 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [68b3db6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:23,499 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5889c3b6
>2012-12-17 09:08:30,046 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [29a7f0f3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:08:33,555 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [670d1744] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:08:33,576 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [670d1744] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4761
>2012-12-17 09:08:33,578 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [670d1744] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4761, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3c51721b
>2012-12-17 09:08:33,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [670d1744] spmStart polling started: taskId = b63f19ee-1920-4163-9bbe-155e59ccfc6c
>2012-12-17 09:08:34,624 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [670d1744] spmStart polling ended: taskId = b63f19ee-1920-4163-9bbe-155e59ccfc6c task status = finished
>2012-12-17 09:08:34,642 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [670d1744] spmStart polling ended, spm status: SPM
>2012-12-17 09:08:34,643 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [670d1744] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b63f19ee-1920-4163-9bbe-155e59ccfc6c), log id: 4472dd81
>2012-12-17 09:08:34,662 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [670d1744] FINISH, HSMClearTaskVDSCommand, log id: 4472dd81
>2012-12-17 09:08:34,662 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [670d1744] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@512c6342, log id: 3c51721b
>2012-12-17 09:08:34,698 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [670d1744] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:08:34,757 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:08:34,757 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 524ffe4e
>2012-12-17 09:08:34,789 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [670d1744] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:08:34,970 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-60) [3299bae4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:08:35,025 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4636d8b4
>2012-12-17 09:08:35,026 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 62fa569c
>2012-12-17 09:08:35,046 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:35,198 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, SpmStopVDSCommand, log id: 62fa569c
>2012-12-17 09:08:35,226 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, ResetIrsVDSCommand, log id: 4636d8b4
>2012-12-17 09:08:35,237 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 282f4ab3
>2012-12-17 09:08:35,237 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:35,237 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 282f4ab3
>2012-12-17 09:08:35,305 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:08:35,316 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4762
>2012-12-17 09:08:35,317 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4762, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 43322053
>2012-12-17 09:08:35,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling started: taskId = cd6172b7-46c9-4ebd-b26f-62fbf7cbcfff
>2012-12-17 09:08:36,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling ended: taskId = cd6172b7-46c9-4ebd-b26f-62fbf7cbcfff task status = finished
>2012-12-17 09:08:36,380 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling ended, spm status: SPM
>2012-12-17 09:08:36,382 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cd6172b7-46c9-4ebd-b26f-62fbf7cbcfff), log id: 638add10
>2012-12-17 09:08:36,400 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMClearTaskVDSCommand, log id: 638add10
>2012-12-17 09:08:36,400 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31b7787e, log id: 43322053
>2012-12-17 09:08:36,527 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:08:36,604 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:36,608 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25d1978e
>2012-12-17 09:08:36,627 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25d1978e
>2012-12-17 09:08:36,627 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 524ffe4e
>2012-12-17 09:08:36,627 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:08:36,628 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6334ff07
>2012-12-17 09:08:36,628 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:36,629 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 76761b4e
>2012-12-17 09:08:36,647 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 76761b4e
>2012-12-17 09:08:36,647 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6334ff07
>2012-12-17 09:08:36,647 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:08:40,227 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:08:46,099 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [371b8996] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:08:46,780 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [480a798f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:08:46,869 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [480a798f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3ea1af92
>2012-12-17 09:08:46,870 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [480a798f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 40f63719
>2012-12-17 09:08:46,894 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [480a798f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:47,061 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [480a798f] FINISH, SpmStopVDSCommand, log id: 40f63719
>2012-12-17 09:08:47,081 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [480a798f] FINISH, ResetIrsVDSCommand, log id: 3ea1af92
>2012-12-17 09:08:47,092 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [480a798f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 431ea8e8
>2012-12-17 09:08:47,092 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [480a798f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:47,092 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [480a798f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 431ea8e8
>2012-12-17 09:08:51,646 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:08:57,143 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [46e0c818] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:08:57,155 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [46e0c818] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4763
>2012-12-17 09:08:57,156 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4763, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5972443f
>2012-12-17 09:08:57,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] spmStart polling started: taskId = a71533a1-221e-4d67-9c7e-bcb26c6574d7
>2012-12-17 09:08:58,199 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] spmStart polling ended: taskId = a71533a1-221e-4d67-9c7e-bcb26c6574d7 task status = finished
>2012-12-17 09:08:58,219 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] spmStart polling ended, spm status: SPM
>2012-12-17 09:08:58,220 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a71533a1-221e-4d67-9c7e-bcb26c6574d7), log id: 79fc1eb3
>2012-12-17 09:08:58,239 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] FINISH, HSMClearTaskVDSCommand, log id: 79fc1eb3
>2012-12-17 09:08:58,239 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1743b856, log id: 5972443f
>2012-12-17 09:08:58,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [46e0c818] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:08:58,332 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [1d8c274e] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:08:58,332 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1203fae8
>2012-12-17 09:08:58,338 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [46e0c818] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:08:58,519 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [779086b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:08:58,727 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4427ee24
>2012-12-17 09:08:58,729 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 53c11695
>2012-12-17 09:08:58,750 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [779086b5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:58,884 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, SpmStopVDSCommand, log id: 53c11695
>2012-12-17 09:08:58,902 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, ResetIrsVDSCommand, log id: 4427ee24
>2012-12-17 09:08:58,914 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3e35fd7f
>2012-12-17 09:08:58,914 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:08:58,914 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3e35fd7f
>2012-12-17 09:08:59,042 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:08:59,055 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4764
>2012-12-17 09:08:59,057 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4764, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2070b779
>2012-12-17 09:08:59,082 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] spmStart polling started: taskId = eb12ad9a-f6de-4f76-b3e0-24eafc4a06d9
>2012-12-17 09:09:00,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] spmStart polling ended: taskId = eb12ad9a-f6de-4f76-b3e0-24eafc4a06d9 task status = finished
>2012-12-17 09:09:00,123 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] spmStart polling ended, spm status: SPM
>2012-12-17 09:09:00,124 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=eb12ad9a-f6de-4f76-b3e0-24eafc4a06d9), log id: 41b3cbe5
>2012-12-17 09:09:00,143 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, HSMClearTaskVDSCommand, log id: 41b3cbe5
>2012-12-17 09:09:00,143 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@30f4916, log id: 2070b779
>2012-12-17 09:09:00,156 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:09:00,170 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:00,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6de2b919
>2012-12-17 09:09:00,193 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6de2b919
>2012-12-17 09:09:00,193 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1203fae8
>2012-12-17 09:09:00,194 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [1d8c274e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:09:00,194 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3fe650ab
>2012-12-17 09:09:00,194 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:00,195 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 32a07141
>2012-12-17 09:09:00,215 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 32a07141
>2012-12-17 09:09:00,215 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3fe650ab
>2012-12-17 09:09:00,215 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [1d8c274e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:09:01,944 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:09:09,085 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [1c12d57c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:09:09,388 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [58ac9a0f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:09:09,453 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4fdddfe3
>2012-12-17 09:09:09,455 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7acd073a
>2012-12-17 09:09:09,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:09,621 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] FINISH, SpmStopVDSCommand, log id: 7acd073a
>2012-12-17 09:09:09,653 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] FINISH, ResetIrsVDSCommand, log id: 4fdddfe3
>2012-12-17 09:09:09,664 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 10ade151
>2012-12-17 09:09:09,664 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [58ac9a0f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:09,664 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 10ade151
>2012-12-17 09:09:12,155 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:09:19,716 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:09:19,727 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4765
>2012-12-17 09:09:19,729 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4765, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 32802a5f
>2012-12-17 09:09:19,750 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] spmStart polling started: taskId = f1fd9523-c772-41bb-8f64-49054fdbaab0
>2012-12-17 09:09:20,771 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] spmStart polling ended: taskId = f1fd9523-c772-41bb-8f64-49054fdbaab0 task status = finished
>2012-12-17 09:09:20,789 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] spmStart polling ended, spm status: SPM
>2012-12-17 09:09:20,791 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f1fd9523-c772-41bb-8f64-49054fdbaab0), log id: 64371594
>2012-12-17 09:09:20,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] FINISH, HSMClearTaskVDSCommand, log id: 64371594
>2012-12-17 09:09:20,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5c69daf2, log id: 32802a5f
>2012-12-17 09:09:21,049 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:09:21,357 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [32d89e5e] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:09:21,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 63722f99
>2012-12-17 09:09:21,478 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:09:21,610 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [1000cccb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:09:21,652 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [1000cccb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 36a4c590
>2012-12-17 09:09:21,653 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1000cccb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6e4df0d6
>2012-12-17 09:09:21,673 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1000cccb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:21,846 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1000cccb] FINISH, SpmStopVDSCommand, log id: 6e4df0d6
>2012-12-17 09:09:21,855 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [1000cccb] FINISH, ResetIrsVDSCommand, log id: 36a4c590
>2012-12-17 09:09:21,865 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [1000cccb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c6fff72
>2012-12-17 09:09:21,866 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1000cccb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:21,866 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [1000cccb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c6fff72
>2012-12-17 09:09:21,918 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [32d89e5e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:09:21,930 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [32d89e5e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4766
>2012-12-17 09:09:21,931 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4766, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 48bcacfa
>2012-12-17 09:09:21,954 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] spmStart polling started: taskId = 2cd466cd-27f4-41b1-a847-e269f21b6f44
>2012-12-17 09:09:22,975 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] spmStart polling ended: taskId = 2cd466cd-27f4-41b1-a847-e269f21b6f44 task status = finished
>2012-12-17 09:09:22,993 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] spmStart polling ended, spm status: SPM
>2012-12-17 09:09:22,994 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2cd466cd-27f4-41b1-a847-e269f21b6f44), log id: 4bc20056
>2012-12-17 09:09:23,012 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, HSMClearTaskVDSCommand, log id: 4bc20056
>2012-12-17 09:09:23,012 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2da434f5, log id: 48bcacfa
>2012-12-17 09:09:23,027 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [32d89e5e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:09:23,054 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:23,059 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 78b88ed2
>2012-12-17 09:09:23,078 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 78b88ed2
>2012-12-17 09:09:23,078 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 63722f99
>2012-12-17 09:09:23,078 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [32d89e5e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:09:23,079 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 98725b2
>2012-12-17 09:09:23,079 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:23,080 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 63f119ed
>2012-12-17 09:09:23,098 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 63f119ed
>2012-12-17 09:09:23,098 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 98725b2
>2012-12-17 09:09:23,098 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [32d89e5e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:09:31,991 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4256c72b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:09:32,109 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [746c8fa6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:09:32,154 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 659d8152
>2012-12-17 09:09:32,156 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1958c16b
>2012-12-17 09:09:32,176 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:32,594 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61acfe5d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:09:32,709 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, SpmStopVDSCommand, log id: 1958c16b
>2012-12-17 09:09:32,728 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, ResetIrsVDSCommand, log id: 659d8152
>2012-12-17 09:09:32,739 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: bc66976
>2012-12-17 09:09:32,739 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [746c8fa6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:32,739 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: bc66976
>2012-12-17 09:09:42,825 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [19e66a45] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:09:42,853 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [19e66a45] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4767
>2012-12-17 09:09:42,855 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4767, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6778ce3f
>2012-12-17 09:09:42,878 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] spmStart polling started: taskId = 5f352215-62a1-42e4-8ad1-793d515ac750
>2012-12-17 09:09:43,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] spmStart polling ended: taskId = 5f352215-62a1-42e4-8ad1-793d515ac750 task status = finished
>2012-12-17 09:09:43,918 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] spmStart polling ended, spm status: SPM
>2012-12-17 09:09:43,919 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5f352215-62a1-42e4-8ad1-793d515ac750), log id: f50d1a1
>2012-12-17 09:09:43,938 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] FINISH, HSMClearTaskVDSCommand, log id: f50d1a1
>2012-12-17 09:09:43,938 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2f2a8e2, log id: 6778ce3f
>2012-12-17 09:09:43,945 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [19e66a45] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:09:43,982 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [75b7a65b] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:09:43,982 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 265f4369
>2012-12-17 09:09:44,013 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [19e66a45] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:09:44,120 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [61bf9ffc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:09:44,177 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 27b3d4e6
>2012-12-17 09:09:44,179 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 55dfa68c
>2012-12-17 09:09:44,199 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:44,328 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] FINISH, SpmStopVDSCommand, log id: 55dfa68c
>2012-12-17 09:09:44,371 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] FINISH, ResetIrsVDSCommand, log id: 27b3d4e6
>2012-12-17 09:09:44,381 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2249bae
>2012-12-17 09:09:44,381 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:44,381 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2249bae
>2012-12-17 09:09:44,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:09:44,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4768
>2012-12-17 09:09:44,438 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4768, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4ff17052
>2012-12-17 09:09:44,461 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling started: taskId = 03d1f9f5-48d7-4385-9ede-e707e7b71f1d
>2012-12-17 09:09:45,483 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling ended: taskId = 03d1f9f5-48d7-4385-9ede-e707e7b71f1d task status = finished
>2012-12-17 09:09:45,501 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling ended, spm status: SPM
>2012-12-17 09:09:45,502 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=03d1f9f5-48d7-4385-9ede-e707e7b71f1d), log id: 72e9658e
>2012-12-17 09:09:45,521 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, HSMClearTaskVDSCommand, log id: 72e9658e
>2012-12-17 09:09:45,521 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@390a8892, log id: 4ff17052
>2012-12-17 09:09:45,880 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:09:45,974 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:45,978 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 72257f37
>2012-12-17 09:09:45,998 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 72257f37
>2012-12-17 09:09:45,998 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 265f4369
>2012-12-17 09:09:45,998 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [75b7a65b] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:09:45,998 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7bc32e6c
>2012-12-17 09:09:45,998 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:46,000 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 67a58f37
>2012-12-17 09:09:46,018 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 67a58f37
>2012-12-17 09:09:46,018 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7bc32e6c
>2012-12-17 09:09:46,018 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [75b7a65b] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:09:53,008 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:09:54,495 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [5dcd105e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:09:54,625 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-34) [7d47f958] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:09:54,732 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 14ff3c7d
>2012-12-17 09:09:54,734 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5c70a8bd
>2012-12-17 09:09:54,754 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:55,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] FINISH, SpmStopVDSCommand, log id: 5c70a8bd
>2012-12-17 09:09:55,923 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] FINISH, ResetIrsVDSCommand, log id: 14ff3c7d
>2012-12-17 09:09:55,934 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 78465a5e
>2012-12-17 09:09:55,934 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7d47f958] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:09:55,934 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 78465a5e
>2012-12-17 09:10:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [7d79337c] Checking autorecoverable hosts
>2012-12-17 09:10:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [7d79337c] Autorecovering 0 hosts
>2012-12-17 09:10:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [7d79337c] Checking autorecoverable hosts done
>2012-12-17 09:10:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [7d79337c] Autorecovering storage domains is disabled, skipping
>2012-12-17 09:10:03,199 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:10:06,025 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [29a7f0f3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:10:06,137 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [29a7f0f3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4769
>2012-12-17 09:10:06,138 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4769, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6dde86c2
>2012-12-17 09:10:06,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] spmStart polling started: taskId = 59672a3a-268e-4d7c-af79-8ed3a72614f3
>2012-12-17 09:10:07,183 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] spmStart polling ended: taskId = 59672a3a-268e-4d7c-af79-8ed3a72614f3 task status = finished
>2012-12-17 09:10:07,201 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] spmStart polling ended, spm status: SPM
>2012-12-17 09:10:07,202 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=59672a3a-268e-4d7c-af79-8ed3a72614f3), log id: 2c297817
>2012-12-17 09:10:07,221 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] FINISH, HSMClearTaskVDSCommand, log id: 2c297817
>2012-12-17 09:10:07,221 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@568b44a7, log id: 6dde86c2
>2012-12-17 09:10:07,243 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [29a7f0f3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:10:07,272 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [436410e6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:10:07,272 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: d13e969
>2012-12-17 09:10:07,302 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [29a7f0f3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:10:07,429 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [75aa641b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:10:07,494 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [75aa641b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 61a6c9f7
>2012-12-17 09:10:07,496 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [75aa641b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2775d5e0
>2012-12-17 09:10:07,515 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [75aa641b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:07,676 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [75aa641b] FINISH, SpmStopVDSCommand, log id: 2775d5e0
>2012-12-17 09:10:07,704 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [75aa641b] FINISH, ResetIrsVDSCommand, log id: 61a6c9f7
>2012-12-17 09:10:07,714 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [75aa641b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 618f161
>2012-12-17 09:10:07,715 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [75aa641b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:07,715 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [75aa641b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 618f161
>2012-12-17 09:10:07,792 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [436410e6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:10:07,821 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [436410e6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4770
>2012-12-17 09:10:07,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4770, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 30c6d720
>2012-12-17 09:10:07,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [436410e6] spmStart polling started: taskId = 076ccd64-1470-4c6d-9db3-21d3b57dd35e
>2012-12-17 09:10:08,868 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [436410e6] spmStart polling ended: taskId = 076ccd64-1470-4c6d-9db3-21d3b57dd35e task status = finished
>2012-12-17 09:10:08,887 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [436410e6] spmStart polling ended, spm status: SPM
>2012-12-17 09:10:08,889 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=076ccd64-1470-4c6d-9db3-21d3b57dd35e), log id: 7c3e312d
>2012-12-17 09:10:08,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, HSMClearTaskVDSCommand, log id: 7c3e312d
>2012-12-17 09:10:08,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@570ec114, log id: 30c6d720
>2012-12-17 09:10:08,924 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [436410e6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:10:08,939 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:08,943 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6ddbceba
>2012-12-17 09:10:08,962 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6ddbceba
>2012-12-17 09:10:08,962 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: d13e969
>2012-12-17 09:10:08,962 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [436410e6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:10:08,962 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 36b31905
>2012-12-17 09:10:08,962 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:08,964 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d51e944
>2012-12-17 09:10:08,983 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d51e944
>2012-12-17 09:10:08,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 36b31905
>2012-12-17 09:10:08,983 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [436410e6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:10:13,372 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:10:17,829 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [505260a3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:10:17,981 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-36) [3ff8505] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:10:18,035 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7dd42974
>2012-12-17 09:10:18,036 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3d5250b5
>2012-12-17 09:10:18,056 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:18,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, SpmStopVDSCommand, log id: 3d5250b5
>2012-12-17 09:10:18,219 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, ResetIrsVDSCommand, log id: 7dd42974
>2012-12-17 09:10:18,230 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3f6e55e
>2012-12-17 09:10:18,230 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:18,230 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3f6e55e
>2012-12-17 09:10:23,606 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:10:28,282 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:10:28,300 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4771
>2012-12-17 09:10:28,301 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4771, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 45b8a6f5
>2012-12-17 09:10:28,324 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] spmStart polling started: taskId = 47fdb2f6-6497-4e9a-b8f7-8b5591f10873
>2012-12-17 09:10:29,346 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] spmStart polling ended: taskId = 47fdb2f6-6497-4e9a-b8f7-8b5591f10873 task status = finished
>2012-12-17 09:10:29,364 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] spmStart polling ended, spm status: SPM
>2012-12-17 09:10:29,366 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=47fdb2f6-6497-4e9a-b8f7-8b5591f10873), log id: 41fc4fb3
>2012-12-17 09:10:29,386 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, HSMClearTaskVDSCommand, log id: 41fc4fb3
>2012-12-17 09:10:29,386 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1bd4929d, log id: 45b8a6f5
>2012-12-17 09:10:29,416 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:10:29,475 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:10:29,475 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1e96bce6
>2012-12-17 09:10:29,545 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:10:29,666 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [492f1d2b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:10:29,734 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [492f1d2b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4076de53
>2012-12-17 09:10:29,735 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [492f1d2b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6e3fd33b
>2012-12-17 09:10:29,755 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [492f1d2b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:30,000 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [492f1d2b] FINISH, SpmStopVDSCommand, log id: 6e3fd33b
>2012-12-17 09:10:30,039 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [492f1d2b] FINISH, ResetIrsVDSCommand, log id: 4076de53
>2012-12-17 09:10:30,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [492f1d2b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 41c64596
>2012-12-17 09:10:30,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [492f1d2b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:30,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [492f1d2b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 41c64596
>2012-12-17 09:10:30,128 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:10:30,162 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4772
>2012-12-17 09:10:30,164 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4772, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4af1ac7d
>2012-12-17 09:10:30,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling started: taskId = 49e076d4-c32b-4887-a4f4-29c59d54d6be
>2012-12-17 09:10:31,211 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended: taskId = 49e076d4-c32b-4887-a4f4-29c59d54d6be task status = finished
>2012-12-17 09:10:31,229 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended, spm status: SPM
>2012-12-17 09:10:31,230 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=49e076d4-c32b-4887-a4f4-29c59d54d6be), log id: 4478235e
>2012-12-17 09:10:31,248 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMClearTaskVDSCommand, log id: 4478235e
>2012-12-17 09:10:31,248 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2b5a1370, log id: 4af1ac7d
>2012-12-17 09:10:31,268 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:10:31,316 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:31,318 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: fc815e0
>2012-12-17 09:10:31,337 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: fc815e0
>2012-12-17 09:10:31,337 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1e96bce6
>2012-12-17 09:10:31,337 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:10:31,337 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7ba4c464
>2012-12-17 09:10:31,337 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:31,339 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: d614025
>2012-12-17 09:10:31,358 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: d614025
>2012-12-17 09:10:31,358 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7ba4c464
>2012-12-17 09:10:31,358 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:10:33,844 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [680d007a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:10:40,158 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [1d6e3277] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:10:41,366 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [14ba2ea4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:10:41,695 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [14ba2ea4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4529cfaa
>2012-12-17 09:10:41,740 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [14ba2ea4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 28a1da6a
>2012-12-17 09:10:41,760 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [14ba2ea4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:41,911 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [14ba2ea4] FINISH, SpmStopVDSCommand, log id: 28a1da6a
>2012-12-17 09:10:41,929 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [14ba2ea4] FINISH, ResetIrsVDSCommand, log id: 4529cfaa
>2012-12-17 09:10:41,946 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [14ba2ea4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2412caca
>2012-12-17 09:10:41,946 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [14ba2ea4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:41,946 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [14ba2ea4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2412caca
>2012-12-17 09:10:44,039 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:10:52,000 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [5d11909d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:10:52,044 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [5d11909d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4773
>2012-12-17 09:10:52,046 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4773, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40099959
>2012-12-17 09:10:52,068 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] spmStart polling started: taskId = 88a6b190-2003-4b24-b4a0-25be1b5ba19f
>2012-12-17 09:10:53,094 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] spmStart polling ended: taskId = 88a6b190-2003-4b24-b4a0-25be1b5ba19f task status = finished
>2012-12-17 09:10:53,113 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] spmStart polling ended, spm status: SPM
>2012-12-17 09:10:53,114 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=88a6b190-2003-4b24-b4a0-25be1b5ba19f), log id: 130279a6
>2012-12-17 09:10:53,133 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] FINISH, HSMClearTaskVDSCommand, log id: 130279a6
>2012-12-17 09:10:53,133 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7808b0a, log id: 40099959
>2012-12-17 09:10:53,149 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [5d11909d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:10:53,231 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [4dd32292] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:10:53,231 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7e34be32
>2012-12-17 09:10:53,267 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [5d11909d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:10:53,460 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-3) [570659e5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:10:53,527 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [570659e5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2e028116
>2012-12-17 09:10:53,529 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [570659e5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: c78982f
>2012-12-17 09:10:53,548 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [570659e5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:53,713 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [570659e5] FINISH, SpmStopVDSCommand, log id: c78982f
>2012-12-17 09:10:53,754 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [570659e5] FINISH, ResetIrsVDSCommand, log id: 2e028116
>2012-12-17 09:10:53,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [570659e5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1028bcfe
>2012-12-17 09:10:53,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [570659e5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:53,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [570659e5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1028bcfe
>2012-12-17 09:10:53,821 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:10:53,836 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4774
>2012-12-17 09:10:53,838 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4774, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2fca5d1e
>2012-12-17 09:10:53,859 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] spmStart polling started: taskId = 21abe0df-6443-400b-91e6-e07a2146e7cf
>2012-12-17 09:10:54,880 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] spmStart polling ended: taskId = 21abe0df-6443-400b-91e6-e07a2146e7cf task status = finished
>2012-12-17 09:10:54,898 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] spmStart polling ended, spm status: SPM
>2012-12-17 09:10:54,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=21abe0df-6443-400b-91e6-e07a2146e7cf), log id: 52f67846
>2012-12-17 09:10:54,918 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, HSMClearTaskVDSCommand, log id: 52f67846
>2012-12-17 09:10:54,919 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@33526637, log id: 2fca5d1e
>2012-12-17 09:10:54,954 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:10:54,966 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:54,971 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: c2707bc
>2012-12-17 09:10:54,990 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: c2707bc
>2012-12-17 09:10:54,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7e34be32
>2012-12-17 09:10:54,990 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [4dd32292] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:10:54,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 14017cd
>2012-12-17 09:10:54,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:10:54,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 96b4081
>2012-12-17 09:10:55,010 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 96b4081
>2012-12-17 09:10:55,010 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 14017cd
>2012-12-17 09:10:55,010 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [4dd32292] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:11:03,868 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [4c00bcb5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:11:04,018 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [3e5f7d32] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:11:04,090 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 70119148
>2012-12-17 09:11:04,092 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 63aaa867
>2012-12-17 09:11:04,112 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:11:04,448 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] FINISH, SpmStopVDSCommand, log id: 63aaa867
>2012-12-17 09:11:04,479 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] FINISH, ResetIrsVDSCommand, log id: 70119148
>2012-12-17 09:11:04,490 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 602175da
>2012-12-17 09:11:04,490 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3e5f7d32] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:11:04,490 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 602175da
>2012-12-17 09:11:04,685 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:11:14,559 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [388f9500] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:11:14,571 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [388f9500] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4775
>2012-12-17 09:11:14,572 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [388f9500] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4775, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 52da58f1
>2012-12-17 09:11:14,594 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [388f9500] spmStart polling started: taskId = 7fe6b14e-b819-41ca-a7a5-9c9dc758b06a
>2012-12-17 09:11:15,616 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [388f9500] spmStart polling ended: taskId = 7fe6b14e-b819-41ca-a7a5-9c9dc758b06a task status = finished
>2012-12-17 09:11:15,635 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [388f9500] spmStart polling ended, spm status: SPM
>2012-12-17 09:11:15,636 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [388f9500] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7fe6b14e-b819-41ca-a7a5-9c9dc758b06a), log id: 378a6c
>2012-12-17 09:11:15,655 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [388f9500] FINISH, HSMClearTaskVDSCommand, log id: 378a6c
>2012-12-17 09:11:15,655 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [388f9500] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e60b890, log id: 52da58f1
>2012-12-17 09:11:16,246 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [388f9500] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:11:16,622 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [63fee526] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:11:16,622 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6d2f3b33
>2012-12-17 09:11:16,679 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [388f9500] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:11:16,776 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-35) [31ec2033] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:11:16,817 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4af22ae5
>2012-12-17 09:11:16,819 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 637f81bd
>2012-12-17 09:11:16,838 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:11:17,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] FINISH, SpmStopVDSCommand, log id: 637f81bd
>2012-12-17 09:11:17,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] FINISH, ResetIrsVDSCommand, log id: 4af22ae5
>2012-12-17 09:11:17,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 380a2afa
>2012-12-17 09:11:17,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [31ec2033] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:11:17,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 380a2afa
>2012-12-17 09:11:17,595 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:11:17,609 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4776
>2012-12-17 09:11:17,611 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4776, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7c9f42e9
>2012-12-17 09:11:17,632 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] spmStart polling started: taskId = 1067e1b4-9c2b-4ed6-973c-42fab28f504f
>2012-12-17 09:11:18,655 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] spmStart polling ended: taskId = 1067e1b4-9c2b-4ed6-973c-42fab28f504f task status = finished
>2012-12-17 09:11:18,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] spmStart polling ended, spm status: SPM
>2012-12-17 09:11:18,675 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1067e1b4-9c2b-4ed6-973c-42fab28f504f), log id: a007a9b
>2012-12-17 09:11:18,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, HSMClearTaskVDSCommand, log id: a007a9b
>2012-12-17 09:11:18,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@25715b3e, log id: 7c9f42e9
>2012-12-17 09:11:18,748 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:11:18,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:11:18,767 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 21cfae1b
>2012-12-17 09:11:18,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 21cfae1b
>2012-12-17 09:11:18,785 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6d2f3b33
>2012-12-17 09:11:18,785 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [63fee526] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:11:18,785 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 49af7ff4
>2012-12-17 09:11:18,785 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:11:18,786 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 43ec14c9
>2012-12-17 09:11:18,805 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 43ec14c9
>2012-12-17 09:11:18,805 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 49af7ff4
>2012-12-17 09:11:18,805 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [63fee526] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:11:25,147 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7e0f01b0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:11:27,653 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [4557a90c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:11:28,047 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [2ed19e27] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:11:28,168 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [2ed19e27] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 63fcd1d2
>2012-12-17 09:11:28,170 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [2ed19e27] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 610168b6
>2012-12-17 09:11:28,191 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [2ed19e27] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:11:28,537 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [2ed19e27] FINISH, SpmStopVDSCommand, log id: 610168b6
>2012-12-17 09:11:28,590 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [2ed19e27] FINISH, ResetIrsVDSCommand, log id: 63fcd1d2
>2012-12-17 09:11:28,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [2ed19e27] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6d162fa0
>2012-12-17 09:11:28,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2ed19e27] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:11:28,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [2ed19e27] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6d162fa0
>2012-12-17 09:11:36,273 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:11:38,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [38fb24e7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:11:38,766 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [38fb24e7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4777
>2012-12-17 09:11:38,767 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4777, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: f2edf66
>2012-12-17 09:11:38,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] spmStart polling started: taskId = 4b4b126d-41eb-4ebe-a356-62c3b9d38cbb
>2012-12-17 09:11:39,814 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] spmStart polling ended: taskId = 4b4b126d-41eb-4ebe-a356-62c3b9d38cbb task status = finished
>2012-12-17 09:11:39,834 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] spmStart polling ended, spm status: SPM
>2012-12-17 09:11:39,836 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4b4b126d-41eb-4ebe-a356-62c3b9d38cbb), log id: 515ba7c5
>2012-12-17 09:11:39,854 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] FINISH, HSMClearTaskVDSCommand, log id: 515ba7c5
>2012-12-17 09:11:39,855 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5863a03d, log id: f2edf66
>2012-12-17 09:11:39,896 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [38fb24e7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:11:39,990 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [39a5c05e] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:11:39,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4ad35453
>2012-12-17 09:11:40,026 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [38fb24e7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:11:40,327 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-56) [733324dc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:11:41,043 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 666d0c86
>2012-12-17 09:11:41,045 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6b6d3b9b
>2012-12-17 09:11:41,065 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [733324dc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:11:41,724 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, SpmStopVDSCommand, log id: 6b6d3b9b
>2012-12-17 09:11:41,819 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, ResetIrsVDSCommand, log id: 666d0c86
>2012-12-17 09:11:41,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7fcd418d
>2012-12-17 09:11:41,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [733324dc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:11:41,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7fcd418d
>2012-12-17 09:11:42,054 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:11:42,076 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4778
>2012-12-17 09:11:42,078 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4778, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6d24de12
>2012-12-17 09:11:42,100 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling started: taskId = 8d264abd-4ea2-4d24-9025-e745e952686f
>2012-12-17 09:11:43,121 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling ended: taskId = 8d264abd-4ea2-4d24-9025-e745e952686f task status = finished
>2012-12-17 09:11:43,139 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling ended, spm status: SPM
>2012-12-17 09:11:43,141 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8d264abd-4ea2-4d24-9025-e745e952686f), log id: 3e4351be
>2012-12-17 09:11:43,159 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, HSMClearTaskVDSCommand, log id: 3e4351be
>2012-12-17 09:11:43,159 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@35abb0e1, log id: 6d24de12
>2012-12-17 09:11:43,234 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:11:43,318 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:11:43,319 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4a8df784
>2012-12-17 09:11:43,339 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4a8df784
>2012-12-17 09:11:43,339 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4ad35453
>2012-12-17 09:11:43,339 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [39a5c05e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:11:43,339 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2e752159
>2012-12-17 09:11:43,339 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:11:43,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1829f3b9
>2012-12-17 09:11:43,359 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1829f3b9
>2012-12-17 09:11:43,359 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2e752159
>2012-12-17 09:11:43,359 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [39a5c05e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:11:46,816 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [94e4f4d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:11:52,140 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [455c512f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:11:52,356 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [13059b24] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:11:52,511 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [13059b24] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5bc5954a
>2012-12-17 09:11:52,513 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [13059b24] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1f51c897
>2012-12-17 09:11:52,533 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [13059b24] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:11:52,679 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [13059b24] FINISH, SpmStopVDSCommand, log id: 1f51c897
>2012-12-17 09:11:52,720 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [13059b24] FINISH, ResetIrsVDSCommand, log id: 5bc5954a
>2012-12-17 09:11:52,731 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [13059b24] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 21724449
>2012-12-17 09:11:52,731 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [13059b24] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:11:52,731 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [13059b24] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 21724449
>2012-12-17 09:11:57,113 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:12:02,819 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:12:02,846 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4779
>2012-12-17 09:12:02,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4779, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 68be638
>2012-12-17 09:12:02,868 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] spmStart polling started: taskId = 593911e9-a3a4-4dfa-97b3-56a8f189bc25
>2012-12-17 09:12:03,890 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] spmStart polling ended: taskId = 593911e9-a3a4-4dfa-97b3-56a8f189bc25 task status = finished
>2012-12-17 09:12:03,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] spmStart polling ended, spm status: SPM
>2012-12-17 09:12:03,909 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=593911e9-a3a4-4dfa-97b3-56a8f189bc25), log id: cfb9b00
>2012-12-17 09:12:03,928 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, HSMClearTaskVDSCommand, log id: cfb9b00
>2012-12-17 09:12:03,928 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5a7d23a8, log id: 68be638
>2012-12-17 09:12:03,946 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:12:04,006 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [1d6be49e] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:12:04,006 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 33c8a1d9
>2012-12-17 09:12:04,025 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:12:04,359 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [786aa7bb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:12:04,652 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6fa59a40
>2012-12-17 09:12:04,654 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 74ad3655
>2012-12-17 09:12:04,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:04,800 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, SpmStopVDSCommand, log id: 74ad3655
>2012-12-17 09:12:04,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, ResetIrsVDSCommand, log id: 6fa59a40
>2012-12-17 09:12:04,921 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1d1e303
>2012-12-17 09:12:04,921 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:04,921 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1d1e303
>2012-12-17 09:12:04,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1d6be49e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:12:05,011 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1d6be49e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4780
>2012-12-17 09:12:05,012 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4780, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7fc380c2
>2012-12-17 09:12:05,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] spmStart polling started: taskId = 6777dc2e-4755-44e6-8ec8-ea7687983784
>2012-12-17 09:12:06,057 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] spmStart polling ended: taskId = 6777dc2e-4755-44e6-8ec8-ea7687983784 task status = finished
>2012-12-17 09:12:06,075 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] spmStart polling ended, spm status: SPM
>2012-12-17 09:12:06,077 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6777dc2e-4755-44e6-8ec8-ea7687983784), log id: 673d694d
>2012-12-17 09:12:06,096 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, HSMClearTaskVDSCommand, log id: 673d694d
>2012-12-17 09:12:06,096 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@70cc732e, log id: 7fc380c2
>2012-12-17 09:12:06,453 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1d6be49e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:12:06,821 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:06,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 27914022
>2012-12-17 09:12:06,844 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 27914022
>2012-12-17 09:12:06,844 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 33c8a1d9
>2012-12-17 09:12:06,844 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [1d6be49e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:12:06,844 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 31098c79
>2012-12-17 09:12:06,844 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:06,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2a77f86
>2012-12-17 09:12:06,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2a77f86
>2012-12-17 09:12:06,864 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 31098c79
>2012-12-17 09:12:06,864 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [1d6be49e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:12:07,374 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [37630ba1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:12:15,062 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5044c757] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:12:15,302 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [26c9f399] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:12:15,487 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5f82833e
>2012-12-17 09:12:15,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3216fe83
>2012-12-17 09:12:15,514 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:15,652 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, SpmStopVDSCommand, log id: 3216fe83
>2012-12-17 09:12:16,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, ResetIrsVDSCommand, log id: 5f82833e
>2012-12-17 09:12:16,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e0bff61
>2012-12-17 09:12:16,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [26c9f399] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:16,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e0bff61
>2012-12-17 09:12:17,828 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:12:26,500 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [6573753] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:12:26,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [6573753] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4781
>2012-12-17 09:12:26,619 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [6573753] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4781, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 69b9d539
>2012-12-17 09:12:26,642 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [6573753] spmStart polling started: taskId = 29a28ae1-8018-47f5-9f54-48460a388266
>2012-12-17 09:12:27,665 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [6573753] spmStart polling ended: taskId = 29a28ae1-8018-47f5-9f54-48460a388266 task status = finished
>2012-12-17 09:12:27,683 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [6573753] spmStart polling ended, spm status: SPM
>2012-12-17 09:12:27,685 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [6573753] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=29a28ae1-8018-47f5-9f54-48460a388266), log id: 7ed923c3
>2012-12-17 09:12:27,704 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [6573753] FINISH, HSMClearTaskVDSCommand, log id: 7ed923c3
>2012-12-17 09:12:27,704 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [6573753] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@61bb1844, log id: 69b9d539
>2012-12-17 09:12:27,716 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [6573753] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:12:27,800 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [69834199] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:12:27,800 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 33bc72bb
>2012-12-17 09:12:27,820 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [6573753] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:12:27,989 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [40c8f45e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:12:28,068 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 67f6b57b
>2012-12-17 09:12:28,070 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1aee0f0e
>2012-12-17 09:12:28,091 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:28,218 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] FINISH, SpmStopVDSCommand, log id: 1aee0f0e
>2012-12-17 09:12:28,233 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] FINISH, ResetIrsVDSCommand, log id: 67f6b57b
>2012-12-17 09:12:28,244 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24a25395
>2012-12-17 09:12:28,244 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [40c8f45e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:28,244 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24a25395
>2012-12-17 09:12:28,306 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:12:28,336 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4782
>2012-12-17 09:12:28,338 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4782, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2c2255d3
>2012-12-17 09:12:28,385 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] spmStart polling started: taskId = 628b1aca-6999-4280-a543-cb7a8efd2dfd
>2012-12-17 09:12:29,407 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] spmStart polling ended: taskId = 628b1aca-6999-4280-a543-cb7a8efd2dfd task status = finished
>2012-12-17 09:12:29,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] spmStart polling ended, spm status: SPM
>2012-12-17 09:12:29,428 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=628b1aca-6999-4280-a543-cb7a8efd2dfd), log id: 185c3ea1
>2012-12-17 09:12:29,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, HSMClearTaskVDSCommand, log id: 185c3ea1
>2012-12-17 09:12:29,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5013c885, log id: 2c2255d3
>2012-12-17 09:12:29,517 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:12:29,548 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:29,552 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2b6a4df9
>2012-12-17 09:12:29,572 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2b6a4df9
>2012-12-17 09:12:29,572 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 33bc72bb
>2012-12-17 09:12:29,572 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [69834199] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:12:29,573 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 50252cb3
>2012-12-17 09:12:29,573 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:29,574 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 76d9c659
>2012-12-17 09:12:29,593 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 76d9c659
>2012-12-17 09:12:29,593 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 50252cb3
>2012-12-17 09:12:29,593 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [69834199] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:12:38,415 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [72736c3b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:12:38,749 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:12:38,895 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4beddd99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:12:39,009 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 58196ba3
>2012-12-17 09:12:39,010 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 66a12ca9
>2012-12-17 09:12:39,030 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:39,804 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] FINISH, SpmStopVDSCommand, log id: 66a12ca9
>2012-12-17 09:12:39,855 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] FINISH, ResetIrsVDSCommand, log id: 58196ba3
>2012-12-17 09:12:39,866 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 401b56a5
>2012-12-17 09:12:39,866 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:39,866 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 401b56a5
>2012-12-17 09:12:49,217 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:12:49,960 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [24cba1b2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:12:49,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [24cba1b2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4783
>2012-12-17 09:12:49,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4783, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4d5b1bf
>2012-12-17 09:12:50,014 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] spmStart polling started: taskId = 196efeca-10cc-469a-a117-96b410c6fc96
>2012-12-17 09:12:51,036 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] spmStart polling ended: taskId = 196efeca-10cc-469a-a117-96b410c6fc96 task status = finished
>2012-12-17 09:12:51,055 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] spmStart polling ended, spm status: SPM
>2012-12-17 09:12:51,056 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=196efeca-10cc-469a-a117-96b410c6fc96), log id: 322a17c2
>2012-12-17 09:12:51,075 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] FINISH, HSMClearTaskVDSCommand, log id: 322a17c2
>2012-12-17 09:12:51,075 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@697ddb48, log id: 4d5b1bf
>2012-12-17 09:12:51,103 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [24cba1b2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:12:51,202 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-17) [3aea96b8] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:12:51,202 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 21aea8d5
>2012-12-17 09:12:51,222 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [24cba1b2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:12:51,537 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [3aebafd8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:12:51,714 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6617d0cc
>2012-12-17 09:12:51,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 558b84aa
>2012-12-17 09:12:51,744 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:51,876 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, SpmStopVDSCommand, log id: 558b84aa
>2012-12-17 09:12:51,965 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, ResetIrsVDSCommand, log id: 6617d0cc
>2012-12-17 09:12:51,978 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 59fbe422
>2012-12-17 09:12:51,978 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3aebafd8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:51,978 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 59fbe422
>2012-12-17 09:12:52,087 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:12:52,119 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4784
>2012-12-17 09:12:52,121 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4784, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6a79eea3
>2012-12-17 09:12:52,142 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] spmStart polling started: taskId = 1e0b0139-1b4b-4a2e-9d49-53f366e033a9
>2012-12-17 09:12:53,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] spmStart polling ended: taskId = 1e0b0139-1b4b-4a2e-9d49-53f366e033a9 task status = finished
>2012-12-17 09:12:53,186 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] spmStart polling ended, spm status: SPM
>2012-12-17 09:12:53,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1e0b0139-1b4b-4a2e-9d49-53f366e033a9), log id: 20987ee8
>2012-12-17 09:12:53,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, HSMClearTaskVDSCommand, log id: 20987ee8
>2012-12-17 09:12:53,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7bb3c20f, log id: 6a79eea3
>2012-12-17 09:12:53,272 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:12:53,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:53,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 31814b62
>2012-12-17 09:12:53,426 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31814b62
>2012-12-17 09:12:53,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 21aea8d5
>2012-12-17 09:12:53,426 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-17) [3aea96b8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:12:53,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1f7c2852
>2012-12-17 09:12:53,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:12:53,428 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 44262c84
>2012-12-17 09:12:53,447 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 44262c84
>2012-12-17 09:12:53,447 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1f7c2852
>2012-12-17 09:12:53,447 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-17) [3aea96b8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:12:59,459 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:13:02,179 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7e0f01b0] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:13:02,508 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [63fee8f1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:13:02,649 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4cf0d994
>2012-12-17 09:13:02,651 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: fc2d205
>2012-12-17 09:13:02,671 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:13:02,877 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] FINISH, SpmStopVDSCommand, log id: fc2d205
>2012-12-17 09:13:03,026 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] FINISH, ResetIrsVDSCommand, log id: 4cf0d994
>2012-12-17 09:13:03,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2a76133e
>2012-12-17 09:13:03,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [63fee8f1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:13:03,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2a76133e
>2012-12-17 09:13:09,883 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [7781d588] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:13:13,214 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:13:13,233 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4785
>2012-12-17 09:13:13,234 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4785, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e887466
>2012-12-17 09:13:13,258 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling started: taskId = e9b1b346-78fd-4602-8f4a-9544e568f59b
>2012-12-17 09:13:14,279 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended: taskId = e9b1b346-78fd-4602-8f4a-9544e568f59b task status = finished
>2012-12-17 09:13:14,297 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended, spm status: SPM
>2012-12-17 09:13:14,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e9b1b346-78fd-4602-8f4a-9544e568f59b), log id: 43df756
>2012-12-17 09:13:14,316 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMClearTaskVDSCommand, log id: 43df756
>2012-12-17 09:13:14,316 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@393147, log id: 5e887466
>2012-12-17 09:13:14,348 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:13:14,418 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:13:14,418 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5330079d
>2012-12-17 09:13:14,488 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:13:14,846 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [66bf54ed] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:13:15,000 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [66bf54ed] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 38f85236
>2012-12-17 09:13:15,002 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [66bf54ed] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 59bf952f
>2012-12-17 09:13:15,023 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [66bf54ed] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:13:15,156 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [66bf54ed] FINISH, SpmStopVDSCommand, log id: 59bf952f
>2012-12-17 09:13:15,213 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [66bf54ed] FINISH, ResetIrsVDSCommand, log id: 38f85236
>2012-12-17 09:13:15,223 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [66bf54ed] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c31b39c
>2012-12-17 09:13:15,224 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [66bf54ed] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:13:15,224 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [66bf54ed] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c31b39c
>2012-12-17 09:13:15,306 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:13:15,322 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4786
>2012-12-17 09:13:15,324 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4786, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5ecb63f6
>2012-12-17 09:13:15,347 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling started: taskId = cb45693f-3eb3-4aad-9289-ab143e955389
>2012-12-17 09:13:16,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended: taskId = cb45693f-3eb3-4aad-9289-ab143e955389 task status = finished
>2012-12-17 09:13:16,386 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended, spm status: SPM
>2012-12-17 09:13:16,388 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cb45693f-3eb3-4aad-9289-ab143e955389), log id: 167dacb5
>2012-12-17 09:13:16,406 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMClearTaskVDSCommand, log id: 167dacb5
>2012-12-17 09:13:16,406 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@14d5aa9c, log id: 5ecb63f6
>2012-12-17 09:13:16,509 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:13:16,558 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:13:16,560 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 78be70e0
>2012-12-17 09:13:16,579 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 78be70e0
>2012-12-17 09:13:16,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5330079d
>2012-12-17 09:13:16,579 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:13:16,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1ee4c00e
>2012-12-17 09:13:16,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:13:16,581 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7d59a879
>2012-12-17 09:13:16,599 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7d59a879
>2012-12-17 09:13:16,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1ee4c00e
>2012-12-17 09:13:16,599 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:13:20,211 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:13:25,408 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:13:26,260 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [3cb7ad26] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:13:26,352 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4575eb09
>2012-12-17 09:13:26,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3bb0ccb2
>2012-12-17 09:13:26,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:13:26,532 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] FINISH, SpmStopVDSCommand, log id: 3bb0ccb2
>2012-12-17 09:13:26,572 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] FINISH, ResetIrsVDSCommand, log id: 4575eb09
>2012-12-17 09:13:26,583 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 18c3de01
>2012-12-17 09:13:26,583 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [3cb7ad26] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:13:26,583 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 18c3de01
>2012-12-17 09:13:31,535 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:13:36,653 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [480a798f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:13:36,741 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [480a798f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4787
>2012-12-17 09:13:36,743 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [480a798f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4787, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 75772e6a
>2012-12-17 09:13:36,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [480a798f] spmStart polling started: taskId = 2b1fa8ab-7a28-4312-bb5e-59dde4310641
>2012-12-17 09:13:37,786 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [480a798f] spmStart polling ended: taskId = 2b1fa8ab-7a28-4312-bb5e-59dde4310641 task status = finished
>2012-12-17 09:13:37,805 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [480a798f] spmStart polling ended, spm status: SPM
>2012-12-17 09:13:37,806 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [480a798f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2b1fa8ab-7a28-4312-bb5e-59dde4310641), log id: 5ce9013
>2012-12-17 09:13:37,825 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [480a798f] FINISH, HSMClearTaskVDSCommand, log id: 5ce9013
>2012-12-17 09:13:37,825 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [480a798f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3c54d25f, log id: 75772e6a
>2012-12-17 09:13:37,868 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [480a798f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:13:37,932 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [1ab078c7] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:13:37,932 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 276c45e2
>2012-12-17 09:13:37,981 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [480a798f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:13:38,163 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [1f8f058a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:13:38,230 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 34a79e1
>2012-12-17 09:13:38,231 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 75f7498e
>2012-12-17 09:13:38,251 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:13:38,399 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] FINISH, SpmStopVDSCommand, log id: 75f7498e
>2012-12-17 09:13:38,443 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] FINISH, ResetIrsVDSCommand, log id: 34a79e1
>2012-12-17 09:13:38,454 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a4175ae
>2012-12-17 09:13:38,454 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [1f8f058a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:13:38,454 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a4175ae
>2012-12-17 09:13:38,539 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:13:38,559 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4788
>2012-12-17 09:13:38,561 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4788, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 20e6b4f7
>2012-12-17 09:13:38,583 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling started: taskId = 6f598347-53cc-4719-9545-0f263ea5553f
>2012-12-17 09:13:39,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling ended: taskId = 6f598347-53cc-4719-9545-0f263ea5553f task status = finished
>2012-12-17 09:13:39,624 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling ended, spm status: SPM
>2012-12-17 09:13:39,625 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6f598347-53cc-4719-9545-0f263ea5553f), log id: 26c44711
>2012-12-17 09:13:39,658 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, HSMClearTaskVDSCommand, log id: 26c44711
>2012-12-17 09:13:39,658 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@55e1d7c3, log id: 20e6b4f7
>2012-12-17 09:13:39,686 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:13:39,728 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:13:39,732 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 33f7c17a
>2012-12-17 09:13:39,752 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 33f7c17a
>2012-12-17 09:13:39,752 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 276c45e2
>2012-12-17 09:13:39,752 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [1ab078c7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:13:39,752 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4ebf4b3d
>2012-12-17 09:13:39,752 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:13:39,754 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6c30c9ad
>2012-12-17 09:13:39,772 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6c30c9ad
>2012-12-17 09:13:39,772 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4ebf4b3d
>2012-12-17 09:13:39,772 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [1ab078c7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:13:41,955 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:13:48,592 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [680d007a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:13:48,809 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [50cb2122] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:13:48,909 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [50cb2122] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1e7d119f
>2012-12-17 09:13:48,910 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [50cb2122] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 271e280c
>2012-12-17 09:13:48,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [50cb2122] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:13:49,097 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [50cb2122] FINISH, SpmStopVDSCommand, log id: 271e280c
>2012-12-17 09:13:49,132 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [50cb2122] FINISH, ResetIrsVDSCommand, log id: 1e7d119f
>2012-12-17 09:13:49,143 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [50cb2122] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2238362e
>2012-12-17 09:13:49,143 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [50cb2122] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:13:49,143 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [50cb2122] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2238362e
>2012-12-17 09:13:52,248 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:13:59,239 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:13:59,250 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4789
>2012-12-17 09:13:59,251 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4789, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1e5dc99c
>2012-12-17 09:13:59,273 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] spmStart polling started: taskId = 93869726-582d-4305-9f04-4e69e3686a13
>2012-12-17 09:14:00,295 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] spmStart polling ended: taskId = 93869726-582d-4305-9f04-4e69e3686a13 task status = finished
>2012-12-17 09:14:00,314 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] spmStart polling ended, spm status: SPM
>2012-12-17 09:14:00,315 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=93869726-582d-4305-9f04-4e69e3686a13), log id: 477eb5c4
>2012-12-17 09:14:00,348 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, HSMClearTaskVDSCommand, log id: 477eb5c4
>2012-12-17 09:14:00,348 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2076d2dd, log id: 1e5dc99c
>2012-12-17 09:14:00,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:14:01,391 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [68b3db6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:14:01,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 43d57c81
>2012-12-17 09:14:01,460 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:14:01,698 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [1598e29f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:14:01,829 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 391fa870
>2012-12-17 09:14:01,831 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 554cf377
>2012-12-17 09:14:01,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:01,986 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, SpmStopVDSCommand, log id: 554cf377
>2012-12-17 09:14:02,036 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, ResetIrsVDSCommand, log id: 391fa870
>2012-12-17 09:14:02,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 20008d8e
>2012-12-17 09:14:02,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:02,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 20008d8e
>2012-12-17 09:14:02,116 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [68b3db6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:14:02,138 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [68b3db6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4790
>2012-12-17 09:14:02,139 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4790, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2244d5b0
>2012-12-17 09:14:02,163 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] spmStart polling started: taskId = d2dc295e-a750-40a4-9522-7419b6b0b812
>2012-12-17 09:14:03,185 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] spmStart polling ended: taskId = d2dc295e-a750-40a4-9522-7419b6b0b812 task status = finished
>2012-12-17 09:14:03,203 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] spmStart polling ended, spm status: SPM
>2012-12-17 09:14:03,204 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d2dc295e-a750-40a4-9522-7419b6b0b812), log id: 7ffaa9c0
>2012-12-17 09:14:03,222 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, HSMClearTaskVDSCommand, log id: 7ffaa9c0
>2012-12-17 09:14:03,223 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@219ebde4, log id: 2244d5b0
>2012-12-17 09:14:03,238 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [68b3db6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:14:03,250 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:03,253 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: c5efca0
>2012-12-17 09:14:03,274 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: c5efca0
>2012-12-17 09:14:03,274 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 43d57c81
>2012-12-17 09:14:03,274 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [68b3db6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:14:03,274 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 301b72b8
>2012-12-17 09:14:03,275 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:03,276 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4ffae45b
>2012-12-17 09:14:03,294 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4ffae45b
>2012-12-17 09:14:03,294 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 301b72b8
>2012-12-17 09:14:03,294 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [68b3db6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:14:12,184 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [32d89e5e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:14:12,394 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [3078e69b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:14:12,667 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [3078e69b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3d80a534
>2012-12-17 09:14:12,668 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [3078e69b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 35dec7ea
>2012-12-17 09:14:12,691 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [3078e69b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:12,866 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [3078e69b] FINISH, SpmStopVDSCommand, log id: 35dec7ea
>2012-12-17 09:14:12,881 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7d3891a6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:14:12,936 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [3078e69b] FINISH, ResetIrsVDSCommand, log id: 3d80a534
>2012-12-17 09:14:12,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [3078e69b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7e43d58f
>2012-12-17 09:14:12,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3078e69b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:12,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [3078e69b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7e43d58f
>2012-12-17 09:14:23,092 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61acfe5d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:14:23,138 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61acfe5d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4791
>2012-12-17 09:14:23,139 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4791, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 70fea370
>2012-12-17 09:14:23,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] spmStart polling started: taskId = 869fd056-c3fb-491c-ae62-2263f44afbcd
>2012-12-17 09:14:24,201 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] spmStart polling ended: taskId = 869fd056-c3fb-491c-ae62-2263f44afbcd task status = finished
>2012-12-17 09:14:24,219 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] spmStart polling ended, spm status: SPM
>2012-12-17 09:14:24,220 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=869fd056-c3fb-491c-ae62-2263f44afbcd), log id: 7f0bfc87
>2012-12-17 09:14:24,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] FINISH, HSMClearTaskVDSCommand, log id: 7f0bfc87
>2012-12-17 09:14:24,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6bf0443a, log id: 70fea370
>2012-12-17 09:14:24,307 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61acfe5d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:14:24,383 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:14:24,384 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 856c21b
>2012-12-17 09:14:24,452 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61acfe5d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:14:24,779 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [11c9636b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:14:24,879 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [11c9636b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 63402df6
>2012-12-17 09:14:24,881 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [11c9636b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 42c6bee4
>2012-12-17 09:14:24,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [11c9636b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:25,290 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [11c9636b] FINISH, SpmStopVDSCommand, log id: 42c6bee4
>2012-12-17 09:14:25,343 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [11c9636b] FINISH, ResetIrsVDSCommand, log id: 63402df6
>2012-12-17 09:14:25,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [11c9636b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: fd0fb57
>2012-12-17 09:14:25,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [11c9636b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:25,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [11c9636b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: fd0fb57
>2012-12-17 09:14:25,596 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:14:25,994 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4792
>2012-12-17 09:14:25,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4792, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5a6d19dc
>2012-12-17 09:14:26,019 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling started: taskId = 77b02066-465c-4f59-b354-742d1955e043
>2012-12-17 09:14:27,043 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling ended: taskId = 77b02066-465c-4f59-b354-742d1955e043 task status = finished
>2012-12-17 09:14:27,061 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling ended, spm status: SPM
>2012-12-17 09:14:27,063 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=77b02066-465c-4f59-b354-742d1955e043), log id: 35421146
>2012-12-17 09:14:27,081 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMClearTaskVDSCommand, log id: 35421146
>2012-12-17 09:14:27,081 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@47c25fb, log id: 5a6d19dc
>2012-12-17 09:14:27,110 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:14:27,166 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:27,168 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5150aff
>2012-12-17 09:14:27,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5150aff
>2012-12-17 09:14:27,187 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 856c21b
>2012-12-17 09:14:27,187 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:14:27,187 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2884edf4
>2012-12-17 09:14:27,188 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:27,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4bd21b70
>2012-12-17 09:14:27,208 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4bd21b70
>2012-12-17 09:14:27,208 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2884edf4
>2012-12-17 09:14:27,208 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:14:33,560 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:14:36,299 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:14:36,777 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [4d69cef1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:14:36,912 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [4d69cef1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 155462b9
>2012-12-17 09:14:36,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [4d69cef1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1700448a
>2012-12-17 09:14:36,935 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [4d69cef1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:37,739 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [4d69cef1] FINISH, SpmStopVDSCommand, log id: 1700448a
>2012-12-17 09:14:37,804 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [4d69cef1] FINISH, ResetIrsVDSCommand, log id: 155462b9
>2012-12-17 09:14:37,814 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [4d69cef1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3a281dca
>2012-12-17 09:14:37,814 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [4d69cef1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:37,814 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [4d69cef1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3a281dca
>2012-12-17 09:14:43,858 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [73cd7795] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:14:47,934 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [46a80193] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:14:47,943 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [46a80193] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4793
>2012-12-17 09:14:47,945 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4793, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 77e3b752
>2012-12-17 09:14:47,982 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [46a80193] spmStart polling started: taskId = af61d983-217d-422c-94e2-9521fc65877c
>2012-12-17 09:14:49,006 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [46a80193] spmStart polling ended: taskId = af61d983-217d-422c-94e2-9521fc65877c task status = finished
>2012-12-17 09:14:49,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [46a80193] spmStart polling ended, spm status: SPM
>2012-12-17 09:14:49,029 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=af61d983-217d-422c-94e2-9521fc65877c), log id: 3e246a24
>2012-12-17 09:14:49,049 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, HSMClearTaskVDSCommand, log id: 3e246a24
>2012-12-17 09:14:49,049 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48b4815b, log id: 77e3b752
>2012-12-17 09:14:49,057 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [46a80193] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:14:49,107 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [1598e29f] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:14:49,107 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 22759ef9
>2012-12-17 09:14:49,119 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [46a80193] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:14:49,228 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [1ebf597f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:14:49,288 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 25bb04fc
>2012-12-17 09:14:49,290 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 72856b38
>2012-12-17 09:14:49,310 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:50,077 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, SpmStopVDSCommand, log id: 72856b38
>2012-12-17 09:14:50,105 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, ResetIrsVDSCommand, log id: 25bb04fc
>2012-12-17 09:14:50,117 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2a620693
>2012-12-17 09:14:50,117 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1ebf597f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:50,117 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2a620693
>2012-12-17 09:14:50,189 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:14:50,203 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4794
>2012-12-17 09:14:50,205 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4794, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 55dc0475
>2012-12-17 09:14:50,227 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] spmStart polling started: taskId = 75e830d7-a31a-4ca1-8ac9-3df7b53abf81
>2012-12-17 09:14:51,250 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] spmStart polling ended: taskId = 75e830d7-a31a-4ca1-8ac9-3df7b53abf81 task status = finished
>2012-12-17 09:14:51,269 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] spmStart polling ended, spm status: SPM
>2012-12-17 09:14:51,271 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=75e830d7-a31a-4ca1-8ac9-3df7b53abf81), log id: 71987d0e
>2012-12-17 09:14:51,290 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, HSMClearTaskVDSCommand, log id: 71987d0e
>2012-12-17 09:14:51,291 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@523a5c65, log id: 55dc0475
>2012-12-17 09:14:51,362 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:14:51,381 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:51,386 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2caac86c
>2012-12-17 09:14:51,413 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2caac86c
>2012-12-17 09:14:51,413 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 22759ef9
>2012-12-17 09:14:51,413 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [1598e29f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:14:51,413 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: e5740d2
>2012-12-17 09:14:51,413 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:14:51,416 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 16d52d8
>2012-12-17 09:14:51,434 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 16d52d8
>2012-12-17 09:14:51,434 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: e5740d2
>2012-12-17 09:14:51,434 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [1598e29f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:14:54,111 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:15:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [94e4f4d] Checking autorecoverable hosts
>2012-12-17 09:15:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [94e4f4d] Autorecovering 0 hosts
>2012-12-17 09:15:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [94e4f4d] Checking autorecoverable hosts done
>2012-12-17 09:15:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [94e4f4d] Autorecovering storage domains is disabled, skipping
>2012-12-17 09:15:00,246 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7d3891a6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:15:00,358 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [50bce313] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:15:00,941 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1000885c
>2012-12-17 09:15:00,943 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3a31d914
>2012-12-17 09:15:00,968 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [50bce313] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:15:01,105 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, SpmStopVDSCommand, log id: 3a31d914
>2012-12-17 09:15:01,594 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, ResetIrsVDSCommand, log id: 1000885c
>2012-12-17 09:15:01,605 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 30a569d9
>2012-12-17 09:15:01,605 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [50bce313] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:15:01,605 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 30a569d9
>2012-12-17 09:15:04,317 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:15:11,823 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:15:11,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4795
>2012-12-17 09:15:11,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4795, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 41e8cfe9
>2012-12-17 09:15:11,947 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling started: taskId = 1713bbb0-ec21-4f86-bd97-dc8765138b74
>2012-12-17 09:15:12,969 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended: taskId = 1713bbb0-ec21-4f86-bd97-dc8765138b74 task status = finished
>2012-12-17 09:15:12,988 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended, spm status: SPM
>2012-12-17 09:15:12,989 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1713bbb0-ec21-4f86-bd97-dc8765138b74), log id: 315b8ccf
>2012-12-17 09:15:13,008 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMClearTaskVDSCommand, log id: 315b8ccf
>2012-12-17 09:15:13,008 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@71798359, log id: 41e8cfe9
>2012-12-17 09:15:13,052 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:15:13,168 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [5231d6f] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:15:13,168 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1c7f49bd
>2012-12-17 09:15:13,193 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:15:13,324 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-15) [e42ef5e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:15:13,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [e42ef5e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 11d58cb8
>2012-12-17 09:15:13,393 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [e42ef5e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 38e24b06
>2012-12-17 09:15:13,413 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [e42ef5e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:15:14,068 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [e42ef5e] FINISH, SpmStopVDSCommand, log id: 38e24b06
>2012-12-17 09:15:14,101 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [e42ef5e] FINISH, ResetIrsVDSCommand, log id: 11d58cb8
>2012-12-17 09:15:14,112 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [e42ef5e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6191f443
>2012-12-17 09:15:14,112 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [e42ef5e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:15:14,112 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [e42ef5e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6191f443
>2012-12-17 09:15:14,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [5231d6f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:15:14,176 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [5231d6f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4796
>2012-12-17 09:15:14,177 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4796, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3c64674d
>2012-12-17 09:15:14,200 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] spmStart polling started: taskId = de0dd12d-7442-4cfe-86d7-dc672c2fbc73
>2012-12-17 09:15:15,221 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] spmStart polling ended: taskId = de0dd12d-7442-4cfe-86d7-dc672c2fbc73 task status = finished
>2012-12-17 09:15:15,239 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] spmStart polling ended, spm status: SPM
>2012-12-17 09:15:15,240 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=de0dd12d-7442-4cfe-86d7-dc672c2fbc73), log id: 500a5409
>2012-12-17 09:15:15,259 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, HSMClearTaskVDSCommand, log id: 500a5409
>2012-12-17 09:15:15,259 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@561343ec, log id: 3c64674d
>2012-12-17 09:15:15,279 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [5231d6f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:15:15,303 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:15:15,307 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 30ceff84
>2012-12-17 09:15:15,326 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 30ceff84
>2012-12-17 09:15:15,327 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1c7f49bd
>2012-12-17 09:15:15,327 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [5231d6f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:15:15,327 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 378603c1
>2012-12-17 09:15:15,327 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:15:15,329 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3e9392c9
>2012-12-17 09:15:15,347 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3e9392c9
>2012-12-17 09:15:15,347 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 378603c1
>2012-12-17 09:15:15,347 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [5231d6f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:15:24,198 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [50cb2122] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:15:24,284 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [49192055] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:15:24,327 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [49192055] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 430c7df9
>2012-12-17 09:15:24,328 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [49192055] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 438cc746
>2012-12-17 09:15:24,348 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [49192055] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:15:24,699 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:15:25,193 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [49192055] FINISH, SpmStopVDSCommand, log id: 438cc746
>2012-12-17 09:15:25,208 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [49192055] FINISH, ResetIrsVDSCommand, log id: 430c7df9
>2012-12-17 09:15:25,219 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [49192055] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4b59f8ac
>2012-12-17 09:15:25,219 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [49192055] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:15:25,219 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [49192055] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4b59f8ac
>2012-12-17 09:15:29,746 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [1940bae2] Running command: LogoutUserCommand internal: false.
>2012-12-17 09:15:35,294 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:15:35,308 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4797
>2012-12-17 09:15:35,309 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4797, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67f1aa2c
>2012-12-17 09:15:35,332 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling started: taskId = 42ca25d3-9536-4de7-91be-4cc049204a87
>2012-12-17 09:15:36,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling ended: taskId = 42ca25d3-9536-4de7-91be-4cc049204a87 task status = finished
>2012-12-17 09:15:36,374 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling ended, spm status: SPM
>2012-12-17 09:15:36,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=42ca25d3-9536-4de7-91be-4cc049204a87), log id: 368e75d9
>2012-12-17 09:15:36,396 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, HSMClearTaskVDSCommand, log id: 368e75d9
>2012-12-17 09:15:36,396 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42db158a, log id: 67f1aa2c
>2012-12-17 09:15:36,454 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:15:36,541 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [7a1c9fc6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:15:36,541 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 495d0c82
>2012-12-17 09:15:36,557 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:15:36,692 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [6c5669f4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:15:36,821 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 77c52c29
>2012-12-17 09:15:36,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 18d4f5a6
>2012-12-17 09:15:36,843 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:15:36,970 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] FINISH, SpmStopVDSCommand, log id: 18d4f5a6
>2012-12-17 09:15:36,998 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] FINISH, ResetIrsVDSCommand, log id: 77c52c29
>2012-12-17 09:15:37,008 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7be18433
>2012-12-17 09:15:37,008 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [6c5669f4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:15:37,008 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7be18433
>2012-12-17 09:15:37,080 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:15:37,110 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4798
>2012-12-17 09:15:37,111 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4798, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 26f9e564
>2012-12-17 09:15:37,132 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling started: taskId = 35d60402-bc3b-489f-add5-9240cfe8a894
>2012-12-17 09:15:38,157 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling ended: taskId = 35d60402-bc3b-489f-add5-9240cfe8a894 task status = finished
>2012-12-17 09:15:38,176 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling ended, spm status: SPM
>2012-12-17 09:15:38,177 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=35d60402-bc3b-489f-add5-9240cfe8a894), log id: 14603702
>2012-12-17 09:15:38,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, HSMClearTaskVDSCommand, log id: 14603702
>2012-12-17 09:15:38,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@38bb6ec1, log id: 26f9e564
>2012-12-17 09:15:38,217 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:15:38,247 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:15:38,251 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 51941608
>2012-12-17 09:15:38,270 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 51941608
>2012-12-17 09:15:38,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 495d0c82
>2012-12-17 09:15:38,271 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [7a1c9fc6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:15:38,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4ed1996e
>2012-12-17 09:15:38,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:15:38,272 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 423d9c0b
>2012-12-17 09:15:38,291 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 423d9c0b
>2012-12-17 09:15:38,291 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4ed1996e
>2012-12-17 09:15:38,291 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [7a1c9fc6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:15:45,136 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1000cccb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:15:47,116 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5bd08b5f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:15:47,260 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [efbb817] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:15:47,330 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [efbb817] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 539fd57
>2012-12-17 09:15:47,332 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [efbb817] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6c38f909
>2012-12-17 09:15:47,352 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [efbb817] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:15:47,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [efbb817] FINISH, SpmStopVDSCommand, log id: 6c38f909
>2012-12-17 09:15:48,001 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [efbb817] FINISH, ResetIrsVDSCommand, log id: 539fd57
>2012-12-17 09:15:48,012 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [efbb817] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b6a8121
>2012-12-17 09:15:48,012 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [efbb817] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:15:48,012 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [efbb817] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b6a8121
>2012-12-17 09:15:56,330 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:15:58,105 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:15:58,115 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4799
>2012-12-17 09:15:58,117 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4799, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6df7dc86
>2012-12-17 09:15:58,139 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling started: taskId = d584a788-8d17-451e-a8e5-0acc8b848d2d
>2012-12-17 09:16:00,188 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling ended: taskId = d584a788-8d17-451e-a8e5-0acc8b848d2d task status = finished
>2012-12-17 09:16:00,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling ended, spm status: SPM
>2012-12-17 09:16:00,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d584a788-8d17-451e-a8e5-0acc8b848d2d), log id: 6da268f9
>2012-12-17 09:16:00,225 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, HSMClearTaskVDSCommand, log id: 6da268f9
>2012-12-17 09:16:00,225 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4199ebaa, log id: 6df7dc86
>2012-12-17 09:16:00,263 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:16:00,309 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [a029edc] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:16:00,309 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4cf61f10
>2012-12-17 09:16:00,324 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:16:01,329 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-97) [3eeaa12a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:16:01,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3bb25634
>2012-12-17 09:16:01,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4305d5e0
>2012-12-17 09:16:01,622 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:01,764 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] FINISH, SpmStopVDSCommand, log id: 4305d5e0
>2012-12-17 09:16:01,802 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] FINISH, ResetIrsVDSCommand, log id: 3bb25634
>2012-12-17 09:16:01,813 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 727efebd
>2012-12-17 09:16:01,813 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [3eeaa12a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:01,813 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 727efebd
>2012-12-17 09:16:01,945 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:16:01,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4800
>2012-12-17 09:16:01,993 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4800, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3785312d
>2012-12-17 09:16:02,017 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] spmStart polling started: taskId = af9d8663-9fde-4f7a-b238-4492ae57d288
>2012-12-17 09:16:03,039 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] spmStart polling ended: taskId = af9d8663-9fde-4f7a-b238-4492ae57d288 task status = finished
>2012-12-17 09:16:03,057 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] spmStart polling ended, spm status: SPM
>2012-12-17 09:16:03,058 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=af9d8663-9fde-4f7a-b238-4492ae57d288), log id: 480ce5de
>2012-12-17 09:16:03,076 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, HSMClearTaskVDSCommand, log id: 480ce5de
>2012-12-17 09:16:03,076 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@46c776c6, log id: 3785312d
>2012-12-17 09:16:03,087 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:16:03,137 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:03,139 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3ec71a4f
>2012-12-17 09:16:03,163 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3ec71a4f
>2012-12-17 09:16:03,163 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4cf61f10
>2012-12-17 09:16:03,163 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [a029edc] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:16:03,163 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1a5aca20
>2012-12-17 09:16:03,163 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:03,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 29926e84
>2012-12-17 09:16:03,184 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 29926e84
>2012-12-17 09:16:03,184 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1a5aca20
>2012-12-17 09:16:03,184 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [a029edc] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:16:06,608 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:16:11,984 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [58c21a2d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:16:12,094 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-76) [67dedb1b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:16:12,225 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3c090cef
>2012-12-17 09:16:12,226 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1ad4d296
>2012-12-17 09:16:12,247 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:12,399 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, SpmStopVDSCommand, log id: 1ad4d296
>2012-12-17 09:16:12,428 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, ResetIrsVDSCommand, log id: 3c090cef
>2012-12-17 09:16:12,439 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 657baea5
>2012-12-17 09:16:12,439 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [67dedb1b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:12,439 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 657baea5
>2012-12-17 09:16:16,820 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:16:22,511 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [58ac9a0f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:16:22,562 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [58ac9a0f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4801
>2012-12-17 09:16:22,563 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4801, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 231f5bd7
>2012-12-17 09:16:22,586 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] spmStart polling started: taskId = c768d2b2-2ab1-4ec8-80a3-825b787f4da1
>2012-12-17 09:16:23,606 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] spmStart polling ended: taskId = c768d2b2-2ab1-4ec8-80a3-825b787f4da1 task status = finished
>2012-12-17 09:16:23,624 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] spmStart polling ended, spm status: SPM
>2012-12-17 09:16:23,626 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c768d2b2-2ab1-4ec8-80a3-825b787f4da1), log id: 5d1c94c2
>2012-12-17 09:16:23,644 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] FINISH, HSMClearTaskVDSCommand, log id: 5d1c94c2
>2012-12-17 09:16:23,644 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7f6a7c4f, log id: 231f5bd7
>2012-12-17 09:16:23,659 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [58ac9a0f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:16:23,725 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-11) [2d4b4fe5] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:16:23,725 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 329468ee
>2012-12-17 09:16:23,741 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [58ac9a0f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:16:23,892 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [5654bc9d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:16:23,975 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 50797fcb
>2012-12-17 09:16:23,977 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1a1f97cb
>2012-12-17 09:16:23,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:24,825 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SpmStopVDSCommand, log id: 1a1f97cb
>2012-12-17 09:16:24,857 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, ResetIrsVDSCommand, log id: 50797fcb
>2012-12-17 09:16:24,867 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b868784
>2012-12-17 09:16:24,867 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:24,867 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b868784
>2012-12-17 09:16:24,936 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:16:24,949 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4802
>2012-12-17 09:16:24,951 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4802, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 596b069f
>2012-12-17 09:16:24,973 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling started: taskId = 455bd6b6-e06c-4b43-ac16-8663aab41a02
>2012-12-17 09:16:25,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling ended: taskId = 455bd6b6-e06c-4b43-ac16-8663aab41a02 task status = finished
>2012-12-17 09:16:26,014 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling ended, spm status: SPM
>2012-12-17 09:16:26,015 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=455bd6b6-e06c-4b43-ac16-8663aab41a02), log id: 12c939e8
>2012-12-17 09:16:26,033 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, HSMClearTaskVDSCommand, log id: 12c939e8
>2012-12-17 09:16:26,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@58f8b7fa, log id: 596b069f
>2012-12-17 09:16:26,155 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:16:26,192 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:26,196 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1b842bfc
>2012-12-17 09:16:26,215 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1b842bfc
>2012-12-17 09:16:26,215 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 329468ee
>2012-12-17 09:16:26,215 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-11) [2d4b4fe5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:16:26,215 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 263bb81
>2012-12-17 09:16:26,215 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:26,216 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4d22488e
>2012-12-17 09:16:26,235 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4d22488e
>2012-12-17 09:16:26,235 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 263bb81
>2012-12-17 09:16:26,235 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-11) [2d4b4fe5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:16:27,122 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:16:34,993 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [436410e6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:16:35,150 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [4cfe9a36] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:16:35,208 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1e06c90d
>2012-12-17 09:16:35,209 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 65633bc6
>2012-12-17 09:16:35,229 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:35,400 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] FINISH, SpmStopVDSCommand, log id: 65633bc6
>2012-12-17 09:16:36,218 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] FINISH, ResetIrsVDSCommand, log id: 1e06c90d
>2012-12-17 09:16:36,228 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 54d2bb05
>2012-12-17 09:16:36,229 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [4cfe9a36] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:36,229 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 54d2bb05
>2012-12-17 09:16:37,489 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:16:46,854 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [1f8f058a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:16:46,880 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [1f8f058a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4803
>2012-12-17 09:16:46,881 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4803, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: ff4f0cd
>2012-12-17 09:16:46,904 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] spmStart polling started: taskId = 4aeafca6-cc76-4cb1-b7fe-45bb3a1229af
>2012-12-17 09:16:47,931 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] spmStart polling ended: taskId = 4aeafca6-cc76-4cb1-b7fe-45bb3a1229af task status = finished
>2012-12-17 09:16:47,950 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] spmStart polling ended, spm status: SPM
>2012-12-17 09:16:47,951 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4aeafca6-cc76-4cb1-b7fe-45bb3a1229af), log id: 653e1d40
>2012-12-17 09:16:47,969 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] FINISH, HSMClearTaskVDSCommand, log id: 653e1d40
>2012-12-17 09:16:47,969 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@50f288ad, log id: ff4f0cd
>2012-12-17 09:16:48,013 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [1f8f058a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:16:48,052 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:16:48,053 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4576353a
>2012-12-17 09:16:48,131 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [1f8f058a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:16:48,375 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [5ecdfb51] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:16:48,495 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6b484266
>2012-12-17 09:16:48,497 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26e048f8
>2012-12-17 09:16:48,517 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:48,703 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] FINISH, SpmStopVDSCommand, log id: 26e048f8
>2012-12-17 09:16:48,729 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] FINISH, ResetIrsVDSCommand, log id: 6b484266
>2012-12-17 09:16:48,740 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 553ba559
>2012-12-17 09:16:48,740 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5ecdfb51] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:48,740 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 553ba559
>2012-12-17 09:16:48,785 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:16:48,796 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4804
>2012-12-17 09:16:48,797 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4804, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4017204f
>2012-12-17 09:16:48,819 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling started: taskId = 9f8a611d-ef58-48d9-b324-eb4c69027521
>2012-12-17 09:16:49,841 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling ended: taskId = 9f8a611d-ef58-48d9-b324-eb4c69027521 task status = finished
>2012-12-17 09:16:49,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling ended, spm status: SPM
>2012-12-17 09:16:49,861 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9f8a611d-ef58-48d9-b324-eb4c69027521), log id: 51e076f1
>2012-12-17 09:16:49,880 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMClearTaskVDSCommand, log id: 51e076f1
>2012-12-17 09:16:49,880 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@67a0f235, log id: 4017204f
>2012-12-17 09:16:49,890 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:16:49,924 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:49,928 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 675af968
>2012-12-17 09:16:49,960 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 675af968
>2012-12-17 09:16:49,960 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4576353a
>2012-12-17 09:16:49,960 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:16:49,960 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e58956a
>2012-12-17 09:16:49,961 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:49,962 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 55164841
>2012-12-17 09:16:49,982 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 55164841
>2012-12-17 09:16:49,982 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e58956a
>2012-12-17 09:16:49,982 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:16:57,879 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [30572eed] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:16:58,827 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [af8a779] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:16:58,947 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [327be49e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:16:59,006 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [327be49e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3f39e29d
>2012-12-17 09:16:59,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [327be49e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 39b8c902
>2012-12-17 09:16:59,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [327be49e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:59,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [327be49e] FINISH, SpmStopVDSCommand, log id: 39b8c902
>2012-12-17 09:16:59,205 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [327be49e] FINISH, ResetIrsVDSCommand, log id: 3f39e29d
>2012-12-17 09:16:59,216 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [327be49e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3a15dcb5
>2012-12-17 09:16:59,216 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [327be49e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:16:59,216 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [327be49e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3a15dcb5
>2012-12-17 09:17:08,129 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [14ba2ea4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:17:09,264 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:17:09,279 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4805
>2012-12-17 09:17:09,280 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4805, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 21303019
>2012-12-17 09:17:09,308 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] spmStart polling started: taskId = 8551fddc-4665-41e6-8742-fa0217fafcaa
>2012-12-17 09:17:10,329 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] spmStart polling ended: taskId = 8551fddc-4665-41e6-8742-fa0217fafcaa task status = finished
>2012-12-17 09:17:10,348 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] spmStart polling ended, spm status: SPM
>2012-12-17 09:17:10,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8551fddc-4665-41e6-8742-fa0217fafcaa), log id: 653f8f50
>2012-12-17 09:17:10,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, HSMClearTaskVDSCommand, log id: 653f8f50
>2012-12-17 09:17:10,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2fda011e, log id: 21303019
>2012-12-17 09:17:10,375 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:17:10,411 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [1ebf597f] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:17:10,411 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5430d7ac
>2012-12-17 09:17:10,789 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:17:12,084 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-81) [5f63eb29] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:17:12,177 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [5f63eb29] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6b8762cf
>2012-12-17 09:17:12,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [5f63eb29] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 584ed778
>2012-12-17 09:17:12,210 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [5f63eb29] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:12,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [5f63eb29] FINISH, SpmStopVDSCommand, log id: 584ed778
>2012-12-17 09:17:12,385 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [5f63eb29] FINISH, ResetIrsVDSCommand, log id: 6b8762cf
>2012-12-17 09:17:12,396 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [5f63eb29] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4cc808a9
>2012-12-17 09:17:12,396 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [5f63eb29] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:12,396 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [5f63eb29] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4cc808a9
>2012-12-17 09:17:12,465 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1ebf597f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:17:12,535 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1ebf597f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4806
>2012-12-17 09:17:12,537 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4806, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 152ff288
>2012-12-17 09:17:12,560 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] spmStart polling started: taskId = e9a2c215-5c17-4089-a5f0-e63ec7af7f32
>2012-12-17 09:17:13,584 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] spmStart polling ended: taskId = e9a2c215-5c17-4089-a5f0-e63ec7af7f32 task status = finished
>2012-12-17 09:17:13,603 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] spmStart polling ended, spm status: SPM
>2012-12-17 09:17:13,604 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e9a2c215-5c17-4089-a5f0-e63ec7af7f32), log id: 72494f20
>2012-12-17 09:17:13,624 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, HSMClearTaskVDSCommand, log id: 72494f20
>2012-12-17 09:17:13,624 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2ae9f87e, log id: 152ff288
>2012-12-17 09:17:13,668 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1ebf597f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:17:13,692 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:13,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 746253d3
>2012-12-17 09:17:13,717 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 746253d3
>2012-12-17 09:17:13,718 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5430d7ac
>2012-12-17 09:17:13,718 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [1ebf597f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:17:13,718 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 636d7d56
>2012-12-17 09:17:13,718 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:13,720 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1b42e7ac
>2012-12-17 09:17:13,739 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1b42e7ac
>2012-12-17 09:17:13,739 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 636d7d56
>2012-12-17 09:17:13,739 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [1ebf597f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:17:18,369 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [75aa641b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:17:22,543 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3078e69b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:17:22,725 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [1240becc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:17:22,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 43eb0c5b
>2012-12-17 09:17:22,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 52832e7
>2012-12-17 09:17:22,786 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1240becc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:22,951 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SpmStopVDSCommand, log id: 52832e7
>2012-12-17 09:17:22,989 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, ResetIrsVDSCommand, log id: 43eb0c5b
>2012-12-17 09:17:22,999 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76b685ae
>2012-12-17 09:17:22,999 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:23,000 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76b685ae
>2012-12-17 09:17:28,586 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:17:33,056 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:17:33,111 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4807
>2012-12-17 09:17:33,113 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4807, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 47685135
>2012-12-17 09:17:33,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] spmStart polling started: taskId = d4f8dba9-02ff-493d-aa62-37394b780ffd
>2012-12-17 09:17:34,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] spmStart polling ended: taskId = d4f8dba9-02ff-493d-aa62-37394b780ffd task status = finished
>2012-12-17 09:17:34,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] spmStart polling ended, spm status: SPM
>2012-12-17 09:17:34,179 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d4f8dba9-02ff-493d-aa62-37394b780ffd), log id: 7fea60ce
>2012-12-17 09:17:34,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, HSMClearTaskVDSCommand, log id: 7fea60ce
>2012-12-17 09:17:34,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7479b5fd, log id: 47685135
>2012-12-17 09:17:34,211 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:17:34,243 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:17:34,243 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 156ba855
>2012-12-17 09:17:34,260 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:17:34,363 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-99) [2a7b65] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:17:34,412 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 44b8746a
>2012-12-17 09:17:34,414 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 24c53e62
>2012-12-17 09:17:34,434 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:34,624 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] FINISH, SpmStopVDSCommand, log id: 24c53e62
>2012-12-17 09:17:34,688 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] FINISH, ResetIrsVDSCommand, log id: 44b8746a
>2012-12-17 09:17:34,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 42aa18c3
>2012-12-17 09:17:34,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [2a7b65] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:34,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 42aa18c3
>2012-12-17 09:17:34,748 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:17:34,763 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4808
>2012-12-17 09:17:34,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4808, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2256e3b9
>2012-12-17 09:17:34,786 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling started: taskId = c037311e-2cff-4bc5-b95d-d7605a169bf0
>2012-12-17 09:17:35,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended: taskId = c037311e-2cff-4bc5-b95d-d7605a169bf0 task status = finished
>2012-12-17 09:17:35,828 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended, spm status: SPM
>2012-12-17 09:17:35,829 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c037311e-2cff-4bc5-b95d-d7605a169bf0), log id: 544577dc
>2012-12-17 09:17:35,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMClearTaskVDSCommand, log id: 544577dc
>2012-12-17 09:17:35,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1cc037bd, log id: 2256e3b9
>2012-12-17 09:17:36,420 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:17:36,655 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:36,657 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 638d4741
>2012-12-17 09:17:36,695 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 638d4741
>2012-12-17 09:17:36,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 156ba855
>2012-12-17 09:17:36,696 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:17:36,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 74bd81f9
>2012-12-17 09:17:36,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:36,697 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 76214961
>2012-12-17 09:17:36,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 76214961
>2012-12-17 09:17:36,716 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 74bd81f9
>2012-12-17 09:17:36,716 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:17:38,807 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:17:44,792 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [37630ba1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:17:44,990 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [95f0e17] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:17:45,084 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [95f0e17] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 51a45297
>2012-12-17 09:17:45,086 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [95f0e17] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3c1f6d48
>2012-12-17 09:17:45,106 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [95f0e17] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:45,310 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [95f0e17] FINISH, SpmStopVDSCommand, log id: 3c1f6d48
>2012-12-17 09:17:45,330 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [95f0e17] FINISH, ResetIrsVDSCommand, log id: 51a45297
>2012-12-17 09:17:45,341 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [95f0e17] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 70f34122
>2012-12-17 09:17:45,341 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [95f0e17] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:45,341 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [95f0e17] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 70f34122
>2012-12-17 09:17:49,017 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:17:55,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7d47f958] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:17:56,105 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7d47f958] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4809
>2012-12-17 09:17:56,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4809, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44563f52
>2012-12-17 09:17:56,139 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] spmStart polling started: taskId = 97e5e958-f92b-4e4f-a830-f3a1b468792e
>2012-12-17 09:17:57,161 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] spmStart polling ended: taskId = 97e5e958-f92b-4e4f-a830-f3a1b468792e task status = finished
>2012-12-17 09:17:57,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] spmStart polling ended, spm status: SPM
>2012-12-17 09:17:57,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=97e5e958-f92b-4e4f-a830-f3a1b468792e), log id: 779f176e
>2012-12-17 09:17:57,215 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] FINISH, HSMClearTaskVDSCommand, log id: 779f176e
>2012-12-17 09:17:57,215 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6dbb741c, log id: 44563f52
>2012-12-17 09:17:57,249 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7d47f958] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:17:57,308 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [1756fb88] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:17:57,308 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e49064d
>2012-12-17 09:17:57,360 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7d47f958] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:17:57,561 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-34) [7e80f448] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:17:57,614 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 742e020f
>2012-12-17 09:17:57,616 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3b7caae7
>2012-12-17 09:17:57,635 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:58,217 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] FINISH, SpmStopVDSCommand, log id: 3b7caae7
>2012-12-17 09:17:58,278 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] FINISH, ResetIrsVDSCommand, log id: 742e020f
>2012-12-17 09:17:58,288 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 56cb197f
>2012-12-17 09:17:58,289 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7e80f448] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:58,289 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 56cb197f
>2012-12-17 09:17:58,404 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:17:58,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4810
>2012-12-17 09:17:58,428 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4810, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44c76623
>2012-12-17 09:17:58,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] spmStart polling started: taskId = 625795c6-6075-4453-808a-4f0010ef2e37
>2012-12-17 09:17:59,470 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] spmStart polling ended: taskId = 625795c6-6075-4453-808a-4f0010ef2e37 task status = finished
>2012-12-17 09:17:59,488 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] spmStart polling ended, spm status: SPM
>2012-12-17 09:17:59,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=625795c6-6075-4453-808a-4f0010ef2e37), log id: 5dae82bd
>2012-12-17 09:17:59,507 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, HSMClearTaskVDSCommand, log id: 5dae82bd
>2012-12-17 09:17:59,508 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6c097657, log id: 44c76623
>2012-12-17 09:17:59,568 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:17:59,598 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:59,599 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: dd6be65
>2012-12-17 09:17:59,618 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: dd6be65
>2012-12-17 09:17:59,618 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e49064d
>2012-12-17 09:17:59,618 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [1756fb88] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:17:59,619 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: d6ddffc
>2012-12-17 09:17:59,619 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:17:59,620 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 616c55f9
>2012-12-17 09:17:59,638 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 616c55f9
>2012-12-17 09:17:59,638 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: d6ddffc
>2012-12-17 09:17:59,638 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [1756fb88] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:18:08,530 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1000cccb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:18:08,706 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:18:08,758 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7da4b87
>2012-12-17 09:18:08,759 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 11be23a5
>2012-12-17 09:18:08,779 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:18:08,897 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] FINISH, SpmStopVDSCommand, log id: 11be23a5
>2012-12-17 09:18:08,950 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] FINISH, ResetIrsVDSCommand, log id: 7da4b87
>2012-12-17 09:18:08,962 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 67c72dd1
>2012-12-17 09:18:08,962 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:18:08,962 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 67c72dd1
>2012-12-17 09:18:09,637 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [94e4f4d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:18:19,029 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3e5f7d32] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:18:19,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3e5f7d32] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4811
>2012-12-17 09:18:19,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4811, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1eae0c7f
>2012-12-17 09:18:19,075 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] spmStart polling started: taskId = 55613c4e-389e-4eb8-a501-cee38816843c
>2012-12-17 09:18:20,096 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] spmStart polling ended: taskId = 55613c4e-389e-4eb8-a501-cee38816843c task status = finished
>2012-12-17 09:18:20,114 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] spmStart polling ended, spm status: SPM
>2012-12-17 09:18:20,153 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=55613c4e-389e-4eb8-a501-cee38816843c), log id: 6280e3f3
>2012-12-17 09:18:20,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] FINISH, HSMClearTaskVDSCommand, log id: 6280e3f3
>2012-12-17 09:18:20,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7bf5990a, log id: 1eae0c7f
>2012-12-17 09:18:20,194 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3e5f7d32] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:18:20,244 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [746c8fa6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:18:20,244 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 98334b7
>2012-12-17 09:18:20,263 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3e5f7d32] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:18:20,392 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [20c0de80] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:18:20,912 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7bea430f
>2012-12-17 09:18:20,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1d9b0b2b
>2012-12-17 09:18:20,934 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:18:21,073 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, SpmStopVDSCommand, log id: 1d9b0b2b
>2012-12-17 09:18:21,211 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, ResetIrsVDSCommand, log id: 7bea430f
>2012-12-17 09:18:21,222 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1578c314
>2012-12-17 09:18:21,222 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [20c0de80] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:18:21,222 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1578c314
>2012-12-17 09:18:21,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [746c8fa6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:18:21,283 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [746c8fa6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4812
>2012-12-17 09:18:21,284 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4812, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e5e2160
>2012-12-17 09:18:21,306 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] spmStart polling started: taskId = 83c3a4d5-895e-47c5-a3ea-fe2be5d44be3
>2012-12-17 09:18:22,327 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] spmStart polling ended: taskId = 83c3a4d5-895e-47c5-a3ea-fe2be5d44be3 task status = finished
>2012-12-17 09:18:22,346 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] spmStart polling ended, spm status: SPM
>2012-12-17 09:18:22,347 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=83c3a4d5-895e-47c5-a3ea-fe2be5d44be3), log id: 49f01313
>2012-12-17 09:18:22,366 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, HSMClearTaskVDSCommand, log id: 49f01313
>2012-12-17 09:18:22,366 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@30746a51, log id: 5e5e2160
>2012-12-17 09:18:22,388 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [746c8fa6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:18:22,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:18:22,401 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 401c7134
>2012-12-17 09:18:22,421 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 401c7134
>2012-12-17 09:18:22,421 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 98334b7
>2012-12-17 09:18:22,421 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [746c8fa6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:18:22,422 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: b4c2dce
>2012-12-17 09:18:22,422 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:18:22,423 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4fd279d6
>2012-12-17 09:18:22,442 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4fd279d6
>2012-12-17 09:18:22,442 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: b4c2dce
>2012-12-17 09:18:22,442 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [746c8fa6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:18:30,016 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:18:31,395 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1d6be49e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:18:31,837 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [2dc4111d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:18:31,975 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3ec6d613
>2012-12-17 09:18:31,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3d4484b
>2012-12-17 09:18:31,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:18:32,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] FINISH, SpmStopVDSCommand, log id: 3d4484b
>2012-12-17 09:18:32,385 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] FINISH, ResetIrsVDSCommand, log id: 3ec6d613
>2012-12-17 09:18:32,396 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 708feda6
>2012-12-17 09:18:32,396 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [2dc4111d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:18:32,396 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 708feda6
>2012-12-17 09:18:40,263 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:18:42,448 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [73cd7795] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:18:42,461 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [73cd7795] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4813
>2012-12-17 09:18:42,462 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4813, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2bf6361
>2012-12-17 09:18:42,484 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] spmStart polling started: taskId = 345bcfb3-b639-4b6a-b53e-704af0fd87f1
>2012-12-17 09:18:43,506 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] spmStart polling ended: taskId = 345bcfb3-b639-4b6a-b53e-704af0fd87f1 task status = finished
>2012-12-17 09:18:43,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] spmStart polling ended, spm status: SPM
>2012-12-17 09:18:43,525 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=345bcfb3-b639-4b6a-b53e-704af0fd87f1), log id: 3c2ca8cc
>2012-12-17 09:18:43,544 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] FINISH, HSMClearTaskVDSCommand, log id: 3c2ca8cc
>2012-12-17 09:18:43,544 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@17fe07f1, log id: 2bf6361
>2012-12-17 09:18:43,558 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [73cd7795] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:18:43,614 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:18:43,614 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 196cd471
>2012-12-17 09:18:43,638 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [73cd7795] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:18:43,792 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [76c72cb4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:18:43,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7006cad7
>2012-12-17 09:18:43,862 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2b9b0a23
>2012-12-17 09:18:43,881 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:18:44,017 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] FINISH, SpmStopVDSCommand, log id: 2b9b0a23
>2012-12-17 09:18:44,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] FINISH, ResetIrsVDSCommand, log id: 7006cad7
>2012-12-17 09:18:44,056 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1e9e4b0b
>2012-12-17 09:18:44,056 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [76c72cb4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:18:44,056 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1e9e4b0b
>2012-12-17 09:18:44,118 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:18:44,135 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4814
>2012-12-17 09:18:44,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4814, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6f5062f6
>2012-12-17 09:18:44,157 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling started: taskId = c8916c00-ccfe-433a-8344-149588056a1c
>2012-12-17 09:18:45,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended: taskId = c8916c00-ccfe-433a-8344-149588056a1c task status = finished
>2012-12-17 09:18:45,196 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended, spm status: SPM
>2012-12-17 09:18:45,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c8916c00-ccfe-433a-8344-149588056a1c), log id: 778d672d
>2012-12-17 09:18:45,215 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMClearTaskVDSCommand, log id: 778d672d
>2012-12-17 09:18:45,216 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6470f8ac, log id: 6f5062f6
>2012-12-17 09:18:45,236 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:18:45,279 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:18:45,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1b05fa49
>2012-12-17 09:18:45,299 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1b05fa49
>2012-12-17 09:18:45,300 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 196cd471
>2012-12-17 09:18:45,300 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:18:45,300 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2572dc7b
>2012-12-17 09:18:45,300 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:18:45,301 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 38921fb7
>2012-12-17 09:18:45,320 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 38921fb7
>2012-12-17 09:18:45,320 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2572dc7b
>2012-12-17 09:18:45,320 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:18:51,405 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:18:54,156 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [75aa641b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:18:54,317 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [32133550] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:18:54,410 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 597b5eac
>2012-12-17 09:18:54,412 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 12a035d0
>2012-12-17 09:18:54,431 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [32133550] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:18:55,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SpmStopVDSCommand, log id: 12a035d0
>2012-12-17 09:18:55,054 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, ResetIrsVDSCommand, log id: 597b5eac
>2012-12-17 09:18:55,065 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6f8b91c7
>2012-12-17 09:18:55,065 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:18:55,065 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6f8b91c7
>2012-12-17 09:19:01,806 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [3cb7ad26] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:19:05,130 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [13059b24] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:19:05,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [13059b24] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4815
>2012-12-17 09:19:05,153 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [13059b24] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4815, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4fd7f7a4
>2012-12-17 09:19:05,175 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [13059b24] spmStart polling started: taskId = 0dcda5e5-0784-4c37-b054-2bb935ba9f67
>2012-12-17 09:19:06,201 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [13059b24] spmStart polling ended: taskId = 0dcda5e5-0784-4c37-b054-2bb935ba9f67 task status = finished
>2012-12-17 09:19:06,219 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [13059b24] spmStart polling ended, spm status: SPM
>2012-12-17 09:19:06,220 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [13059b24] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0dcda5e5-0784-4c37-b054-2bb935ba9f67), log id: 717e54d
>2012-12-17 09:19:06,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [13059b24] FINISH, HSMClearTaskVDSCommand, log id: 717e54d
>2012-12-17 09:19:06,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [13059b24] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31cf25de, log id: 4fd7f7a4
>2012-12-17 09:19:06,325 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [13059b24] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:19:06,404 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [3ff8505] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:19:06,404 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 422f053c
>2012-12-17 09:19:06,461 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [13059b24] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:19:06,755 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [1091267a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:19:06,820 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [1091267a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3d89a5b5
>2012-12-17 09:19:06,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [1091267a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 65cff4a
>2012-12-17 09:19:06,842 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [1091267a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:07,023 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [1091267a] FINISH, SpmStopVDSCommand, log id: 65cff4a
>2012-12-17 09:19:07,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [1091267a] FINISH, ResetIrsVDSCommand, log id: 3d89a5b5
>2012-12-17 09:19:07,048 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [1091267a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5d35054b
>2012-12-17 09:19:07,048 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [1091267a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:07,048 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [1091267a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5d35054b
>2012-12-17 09:19:07,125 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:19:07,153 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4816
>2012-12-17 09:19:07,155 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4816, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2637f7d8
>2012-12-17 09:19:07,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] spmStart polling started: taskId = da3593b6-fd6c-497a-b826-551485e178bf
>2012-12-17 09:19:08,200 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] spmStart polling ended: taskId = da3593b6-fd6c-497a-b826-551485e178bf task status = finished
>2012-12-17 09:19:08,218 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] spmStart polling ended, spm status: SPM
>2012-12-17 09:19:08,220 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=da3593b6-fd6c-497a-b826-551485e178bf), log id: 128ee247
>2012-12-17 09:19:08,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, HSMClearTaskVDSCommand, log id: 128ee247
>2012-12-17 09:19:08,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@62a9acd5, log id: 2637f7d8
>2012-12-17 09:19:08,251 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:19:08,263 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:08,267 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3390cc8c
>2012-12-17 09:19:08,286 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3390cc8c
>2012-12-17 09:19:08,286 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 422f053c
>2012-12-17 09:19:08,286 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [3ff8505] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:19:08,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1d78af32
>2012-12-17 09:19:08,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:08,288 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 65196d6a
>2012-12-17 09:19:08,306 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 65196d6a
>2012-12-17 09:19:08,306 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1d78af32
>2012-12-17 09:19:08,307 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [3ff8505] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:19:12,042 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:19:17,170 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [492f1d2b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:19:17,303 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [1513ca5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:19:17,450 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 25089593
>2012-12-17 09:19:17,451 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5ba1e543
>2012-12-17 09:19:17,471 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:17,618 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] FINISH, SpmStopVDSCommand, log id: 5ba1e543
>2012-12-17 09:19:17,678 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] FINISH, ResetIrsVDSCommand, log id: 25089593
>2012-12-17 09:19:17,689 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 25b44d10
>2012-12-17 09:19:17,689 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [1513ca5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:17,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 25b44d10
>2012-12-17 09:19:22,263 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:19:27,756 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:19:27,784 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4817
>2012-12-17 09:19:27,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [71efd086] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4817, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7154625c
>2012-12-17 09:19:27,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [71efd086] spmStart polling started: taskId = 22c829c2-c0b1-4633-b6d4-7950cc4c77df
>2012-12-17 09:19:28,829 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [71efd086] spmStart polling ended: taskId = 22c829c2-c0b1-4633-b6d4-7950cc4c77df task status = finished
>2012-12-17 09:19:28,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [71efd086] spmStart polling ended, spm status: SPM
>2012-12-17 09:19:28,848 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [71efd086] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=22c829c2-c0b1-4633-b6d4-7950cc4c77df), log id: 36817ba
>2012-12-17 09:19:28,867 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [71efd086] FINISH, HSMClearTaskVDSCommand, log id: 36817ba
>2012-12-17 09:19:28,867 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [71efd086] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@26edb615, log id: 7154625c
>2012-12-17 09:19:28,879 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:19:28,933 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [4c0a2fb5] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:19:28,934 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3c35f8c7
>2012-12-17 09:19:28,964 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:19:29,058 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-51) [1cfebcc3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:19:29,111 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 22eb571b
>2012-12-17 09:19:29,112 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 365064dd
>2012-12-17 09:19:29,132 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:29,753 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] FINISH, SpmStopVDSCommand, log id: 365064dd
>2012-12-17 09:19:29,767 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] FINISH, ResetIrsVDSCommand, log id: 22eb571b
>2012-12-17 09:19:29,778 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c226ee4
>2012-12-17 09:19:29,778 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [1cfebcc3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:29,778 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c226ee4
>2012-12-17 09:19:29,840 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:19:29,866 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4818
>2012-12-17 09:19:29,867 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4818, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 60e3e146
>2012-12-17 09:19:29,888 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] spmStart polling started: taskId = b67380cb-25a9-4a7a-abed-a82766cf28aa
>2012-12-17 09:19:30,909 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] spmStart polling ended: taskId = b67380cb-25a9-4a7a-abed-a82766cf28aa task status = finished
>2012-12-17 09:19:30,927 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] spmStart polling ended, spm status: SPM
>2012-12-17 09:19:30,929 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b67380cb-25a9-4a7a-abed-a82766cf28aa), log id: 6ce6e59a
>2012-12-17 09:19:30,947 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, HSMClearTaskVDSCommand, log id: 6ce6e59a
>2012-12-17 09:19:30,947 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2aec1951, log id: 60e3e146
>2012-12-17 09:19:31,017 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:19:31,198 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:31,202 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3bd41e9f
>2012-12-17 09:19:31,222 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3bd41e9f
>2012-12-17 09:19:31,222 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3c35f8c7
>2012-12-17 09:19:31,222 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [4c0a2fb5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:19:31,222 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 63267768
>2012-12-17 09:19:31,222 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:31,224 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 79e192a4
>2012-12-17 09:19:31,244 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 79e192a4
>2012-12-17 09:19:31,244 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 63267768
>2012-12-17 09:19:31,244 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [4c0a2fb5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:19:32,547 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7e80f448] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:19:39,892 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [495d106] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:19:40,029 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-80) [6c899ac8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:19:40,070 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7acc6f12
>2012-12-17 09:19:40,072 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 339b65d5
>2012-12-17 09:19:40,092 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:40,272 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, SpmStopVDSCommand, log id: 339b65d5
>2012-12-17 09:19:40,279 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, ResetIrsVDSCommand, log id: 7acc6f12
>2012-12-17 09:19:40,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5608f009
>2012-12-17 09:19:40,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:40,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5608f009
>2012-12-17 09:19:42,761 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:19:50,352 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:19:50,413 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4819
>2012-12-17 09:19:50,415 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4819, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: cad9463
>2012-12-17 09:19:50,441 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] spmStart polling started: taskId = fc11cb78-5cbe-4ad5-b1e8-afbb833a9ad5
>2012-12-17 09:19:51,462 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] spmStart polling ended: taskId = fc11cb78-5cbe-4ad5-b1e8-afbb833a9ad5 task status = finished
>2012-12-17 09:19:51,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] spmStart polling ended, spm status: SPM
>2012-12-17 09:19:51,482 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=fc11cb78-5cbe-4ad5-b1e8-afbb833a9ad5), log id: 6325fbe5
>2012-12-17 09:19:51,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, HSMClearTaskVDSCommand, log id: 6325fbe5
>2012-12-17 09:19:51,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@52bcaa3d, log id: cad9463
>2012-12-17 09:19:51,605 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:19:51,729 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [25c7cf42] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:19:51,729 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5e078d40
>2012-12-17 09:19:51,745 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:19:51,925 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [223c8c2d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:19:52,016 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 8d8eebd
>2012-12-17 09:19:52,017 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6b25632b
>2012-12-17 09:19:52,041 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:52,291 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] FINISH, SpmStopVDSCommand, log id: 6b25632b
>2012-12-17 09:19:52,299 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] FINISH, ResetIrsVDSCommand, log id: 8d8eebd
>2012-12-17 09:19:52,310 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: b9049e
>2012-12-17 09:19:52,310 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [223c8c2d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:52,310 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: b9049e
>2012-12-17 09:19:52,367 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [25c7cf42] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:19:52,386 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [25c7cf42] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4820
>2012-12-17 09:19:52,388 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4820, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 54384dfd
>2012-12-17 09:19:52,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] spmStart polling started: taskId = 48af515d-803f-4965-a8bb-bb04060012ea
>2012-12-17 09:19:53,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] spmStart polling ended: taskId = 48af515d-803f-4965-a8bb-bb04060012ea task status = finished
>2012-12-17 09:19:53,447 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] spmStart polling ended, spm status: SPM
>2012-12-17 09:19:53,448 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=48af515d-803f-4965-a8bb-bb04060012ea), log id: f5fa7b5
>2012-12-17 09:19:53,466 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, HSMClearTaskVDSCommand, log id: f5fa7b5
>2012-12-17 09:19:53,467 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@50fb63f5, log id: 54384dfd
>2012-12-17 09:19:53,493 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [25c7cf42] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:19:53,507 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:53,512 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 40b5a4a2
>2012-12-17 09:19:53,531 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 40b5a4a2
>2012-12-17 09:19:53,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5e078d40
>2012-12-17 09:19:53,531 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [25c7cf42] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:19:53,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5a3bb291
>2012-12-17 09:19:53,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:19:53,532 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 73dadf5e
>2012-12-17 09:19:53,551 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 73dadf5e
>2012-12-17 09:19:53,551 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5a3bb291
>2012-12-17 09:19:53,551 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [25c7cf42] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:20:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [7f190dc3] Checking autorecoverable hosts
>2012-12-17 09:20:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [7f190dc3] Autorecovering 0 hosts
>2012-12-17 09:20:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [7f190dc3] Checking autorecoverable hosts done
>2012-12-17 09:20:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [7f190dc3] Autorecovering storage domains is disabled, skipping
>2012-12-17 09:20:02,439 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [5231d6f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:20:02,540 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [4f973759] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:20:02,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [4f973759] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 696bd047
>2012-12-17 09:20:02,583 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [4f973759] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 68566845
>2012-12-17 09:20:02,603 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [4f973759] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:02,794 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [4f973759] FINISH, SpmStopVDSCommand, log id: 68566845
>2012-12-17 09:20:02,816 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [4f973759] FINISH, ResetIrsVDSCommand, log id: 696bd047
>2012-12-17 09:20:02,827 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [4f973759] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5367f7cb
>2012-12-17 09:20:02,827 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4f973759] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:02,827 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [4f973759] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5367f7cb
>2012-12-17 09:20:03,159 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [1513ca5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:20:12,895 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:20:12,924 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4821
>2012-12-17 09:20:12,926 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4821, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 76022377
>2012-12-17 09:20:12,948 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] spmStart polling started: taskId = 71b16ea4-42ed-4b4f-9ee2-b7c87953add2
>2012-12-17 09:20:13,969 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] spmStart polling ended: taskId = 71b16ea4-42ed-4b4f-9ee2-b7c87953add2 task status = finished
>2012-12-17 09:20:13,987 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] spmStart polling ended, spm status: SPM
>2012-12-17 09:20:13,988 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=71b16ea4-42ed-4b4f-9ee2-b7c87953add2), log id: 30d7e649
>2012-12-17 09:20:14,006 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, HSMClearTaskVDSCommand, log id: 30d7e649
>2012-12-17 09:20:14,006 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2e4fdbac, log id: 76022377
>2012-12-17 09:20:14,018 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:20:14,049 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [39739d98] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:20:14,049 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2c06e302
>2012-12-17 09:20:14,080 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:20:14,241 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-85) [e893ae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:20:14,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4ffb02dd
>2012-12-17 09:20:14,289 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: b3b9fc1
>2012-12-17 09:20:14,309 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [e893ae] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:14,568 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, SpmStopVDSCommand, log id: b3b9fc1
>2012-12-17 09:20:14,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, ResetIrsVDSCommand, log id: 4ffb02dd
>2012-12-17 09:20:14,592 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 431c5528
>2012-12-17 09:20:14,592 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:14,592 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 431c5528
>2012-12-17 09:20:14,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:20:14,703 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4822
>2012-12-17 09:20:14,704 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4822, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 23d1642d
>2012-12-17 09:20:14,727 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] spmStart polling started: taskId = e269c584-d23f-4208-b90a-498b15abfe25
>2012-12-17 09:20:15,749 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] spmStart polling ended: taskId = e269c584-d23f-4208-b90a-498b15abfe25 task status = finished
>2012-12-17 09:20:15,773 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] spmStart polling ended, spm status: SPM
>2012-12-17 09:20:15,775 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e269c584-d23f-4208-b90a-498b15abfe25), log id: 3ff5e02b
>2012-12-17 09:20:15,794 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, HSMClearTaskVDSCommand, log id: 3ff5e02b
>2012-12-17 09:20:15,794 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@275047db, log id: 23d1642d
>2012-12-17 09:20:16,215 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:20:16,392 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:16,397 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 236fbbb4
>2012-12-17 09:20:16,417 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 236fbbb4
>2012-12-17 09:20:16,417 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2c06e302
>2012-12-17 09:20:16,417 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [39739d98] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:20:16,418 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7c8c3898
>2012-12-17 09:20:16,418 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:16,419 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4492fe95
>2012-12-17 09:20:16,437 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4492fe95
>2012-12-17 09:20:16,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7c8c3898
>2012-12-17 09:20:16,437 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [39739d98] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:20:23,496 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [6c5669f4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:20:24,776 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1ebf597f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:20:24,880 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [3a1c2706] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:20:24,940 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2aec6e04
>2012-12-17 09:20:24,943 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6be35fa3
>2012-12-17 09:20:24,966 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:25,103 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, SpmStopVDSCommand, log id: 6be35fa3
>2012-12-17 09:20:25,139 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, ResetIrsVDSCommand, log id: 2aec6e04
>2012-12-17 09:20:25,151 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 713790b3
>2012-12-17 09:20:25,151 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:25,151 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 713790b3
>2012-12-17 09:20:33,738 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:20:35,214 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [4cfe9a36] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:20:35,253 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [4cfe9a36] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4823
>2012-12-17 09:20:35,254 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4823, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 45df0eb0
>2012-12-17 09:20:35,277 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] spmStart polling started: taskId = cdcfd788-7350-4642-8159-e4b13ab96a02
>2012-12-17 09:20:36,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] spmStart polling ended: taskId = cdcfd788-7350-4642-8159-e4b13ab96a02 task status = finished
>2012-12-17 09:20:36,316 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] spmStart polling ended, spm status: SPM
>2012-12-17 09:20:36,318 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cdcfd788-7350-4642-8159-e4b13ab96a02), log id: 78b49e9d
>2012-12-17 09:20:36,336 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] FINISH, HSMClearTaskVDSCommand, log id: 78b49e9d
>2012-12-17 09:20:36,336 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1a199354, log id: 45df0eb0
>2012-12-17 09:20:36,370 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [4cfe9a36] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:20:36,420 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [50bce313] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:20:36,420 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5188e1b3
>2012-12-17 09:20:36,444 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [4cfe9a36] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:20:36,539 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [7f36fad2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:20:36,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [7f36fad2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 28f6d03e
>2012-12-17 09:20:36,581 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [7f36fad2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6093ab93
>2012-12-17 09:20:36,600 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [7f36fad2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:37,088 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [7f36fad2] FINISH, SpmStopVDSCommand, log id: 6093ab93
>2012-12-17 09:20:37,103 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [7f36fad2] FINISH, ResetIrsVDSCommand, log id: 28f6d03e
>2012-12-17 09:20:37,114 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [7f36fad2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 70e1dcf3
>2012-12-17 09:20:37,114 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [7f36fad2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:37,114 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [7f36fad2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 70e1dcf3
>2012-12-17 09:20:37,189 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [50bce313] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:20:37,223 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [50bce313] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4824
>2012-12-17 09:20:37,224 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4824, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b79778a
>2012-12-17 09:20:37,247 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [50bce313] spmStart polling started: taskId = e99780cd-1cab-4fd3-8914-f433399592b7
>2012-12-17 09:20:38,268 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [50bce313] spmStart polling ended: taskId = e99780cd-1cab-4fd3-8914-f433399592b7 task status = finished
>2012-12-17 09:20:38,286 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [50bce313] spmStart polling ended, spm status: SPM
>2012-12-17 09:20:38,288 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e99780cd-1cab-4fd3-8914-f433399592b7), log id: 38c43cf5
>2012-12-17 09:20:38,306 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, HSMClearTaskVDSCommand, log id: 38c43cf5
>2012-12-17 09:20:38,306 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5b7e15f0, log id: 6b79778a
>2012-12-17 09:20:38,320 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [50bce313] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:20:38,351 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:38,352 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 780e222f
>2012-12-17 09:20:38,372 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 780e222f
>2012-12-17 09:20:38,372 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5188e1b3
>2012-12-17 09:20:38,372 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [50bce313] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:20:38,372 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4c21876e
>2012-12-17 09:20:38,372 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:38,374 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4cc4e342
>2012-12-17 09:20:38,393 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4cc4e342
>2012-12-17 09:20:38,393 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4c21876e
>2012-12-17 09:20:38,393 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [50bce313] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:20:43,934 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [2a7b65] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:20:47,234 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:20:47,380 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-26) [2a365c8d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:20:47,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6175dbc8
>2012-12-17 09:20:47,482 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1d43046f
>2012-12-17 09:20:47,502 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:47,629 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] FINISH, SpmStopVDSCommand, log id: 1d43046f
>2012-12-17 09:20:47,659 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] FINISH, ResetIrsVDSCommand, log id: 6175dbc8
>2012-12-17 09:20:47,670 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 397d4c77
>2012-12-17 09:20:47,670 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2a365c8d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:47,670 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 397d4c77
>2012-12-17 09:20:54,187 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:20:57,720 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [30572eed] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:20:57,734 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [30572eed] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4825
>2012-12-17 09:20:57,736 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [30572eed] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4825, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4e3714c7
>2012-12-17 09:20:57,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [30572eed] spmStart polling started: taskId = d6c5ae17-c986-49a5-87d7-cb94893128d7
>2012-12-17 09:20:58,779 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [30572eed] spmStart polling ended: taskId = d6c5ae17-c986-49a5-87d7-cb94893128d7 task status = finished
>2012-12-17 09:20:58,797 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [30572eed] spmStart polling ended, spm status: SPM
>2012-12-17 09:20:58,799 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [30572eed] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d6c5ae17-c986-49a5-87d7-cb94893128d7), log id: 186e2179
>2012-12-17 09:20:58,817 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [30572eed] FINISH, HSMClearTaskVDSCommand, log id: 186e2179
>2012-12-17 09:20:58,817 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [30572eed] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@66579b76, log id: 4e3714c7
>2012-12-17 09:20:58,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [30572eed] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:20:58,885 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [67dedb1b] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:20:58,885 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 9abcd85
>2012-12-17 09:20:58,914 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [30572eed] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:20:59,032 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [f37b622] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:20:59,094 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [f37b622] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7707f73a
>2012-12-17 09:20:59,096 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [f37b622] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 42ca738
>2012-12-17 09:20:59,117 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [f37b622] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:59,245 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [f37b622] FINISH, SpmStopVDSCommand, log id: 42ca738
>2012-12-17 09:20:59,256 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [f37b622] FINISH, ResetIrsVDSCommand, log id: 7707f73a
>2012-12-17 09:20:59,269 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [f37b622] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76c3f0d5
>2012-12-17 09:20:59,269 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [f37b622] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:20:59,269 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [f37b622] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76c3f0d5
>2012-12-17 09:20:59,341 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [67dedb1b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:20:59,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [67dedb1b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4826
>2012-12-17 09:20:59,370 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4826, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 12673c7e
>2012-12-17 09:20:59,397 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] spmStart polling started: taskId = edeed2a1-5b68-48a6-943e-e15a5fb67f00
>2012-12-17 09:21:00,417 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] spmStart polling ended: taskId = edeed2a1-5b68-48a6-943e-e15a5fb67f00 task status = finished
>2012-12-17 09:21:00,436 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] spmStart polling ended, spm status: SPM
>2012-12-17 09:21:00,437 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=edeed2a1-5b68-48a6-943e-e15a5fb67f00), log id: 628cde93
>2012-12-17 09:21:00,456 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, HSMClearTaskVDSCommand, log id: 628cde93
>2012-12-17 09:21:00,456 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e68b771, log id: 12673c7e
>2012-12-17 09:21:00,886 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [67dedb1b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:21:01,166 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:01,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 537cf64
>2012-12-17 09:21:01,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 537cf64
>2012-12-17 09:21:01,188 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 9abcd85
>2012-12-17 09:21:01,188 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [67dedb1b] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:21:01,188 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1310b0e9
>2012-12-17 09:21:01,188 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:01,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 41534c19
>2012-12-17 09:21:01,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 41534c19
>2012-12-17 09:21:01,207 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1310b0e9
>2012-12-17 09:21:01,207 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [67dedb1b] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:21:04,370 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:21:09,401 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [66bf54ed] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:21:09,556 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [48bd7fc3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:21:09,605 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17a329a5
>2012-12-17 09:21:09,607 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 73fa0010
>2012-12-17 09:21:09,626 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:09,763 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] FINISH, SpmStopVDSCommand, log id: 73fa0010
>2012-12-17 09:21:09,774 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] FINISH, ResetIrsVDSCommand, log id: 17a329a5
>2012-12-17 09:21:09,785 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c0cb809
>2012-12-17 09:21:09,785 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [48bd7fc3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:09,785 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c0cb809
>2012-12-17 09:21:14,685 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [733324dc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:21:19,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:21:19,851 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4827
>2012-12-17 09:21:19,853 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4827, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 33431252
>2012-12-17 09:21:19,874 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] spmStart polling started: taskId = 8db4002a-d19b-40a5-9417-90169acce97c
>2012-12-17 09:21:20,894 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] spmStart polling ended: taskId = 8db4002a-d19b-40a5-9417-90169acce97c task status = finished
>2012-12-17 09:21:20,913 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] spmStart polling ended, spm status: SPM
>2012-12-17 09:21:20,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8db4002a-d19b-40a5-9417-90169acce97c), log id: 7f8953e1
>2012-12-17 09:21:20,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, HSMClearTaskVDSCommand, log id: 7f8953e1
>2012-12-17 09:21:20,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@10e21a31, log id: 33431252
>2012-12-17 09:21:21,144 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:21:21,960 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1240becc] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:21:21,961 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: aa4a356
>2012-12-17 09:21:22,000 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:21:22,148 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-75) [c8c45fe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:21:22,210 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 50d46249
>2012-12-17 09:21:22,212 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5d07bca4
>2012-12-17 09:21:22,233 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:22,350 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] FINISH, SpmStopVDSCommand, log id: 5d07bca4
>2012-12-17 09:21:22,386 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] FINISH, ResetIrsVDSCommand, log id: 50d46249
>2012-12-17 09:21:22,397 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 602665c5
>2012-12-17 09:21:22,397 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [c8c45fe] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:22,397 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 602665c5
>2012-12-17 09:21:22,452 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:21:22,485 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4828
>2012-12-17 09:21:22,487 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4828, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7651981b
>2012-12-17 09:21:22,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling started: taskId = 25511a55-284e-406f-ae19-8ed081354548
>2012-12-17 09:21:23,530 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling ended: taskId = 25511a55-284e-406f-ae19-8ed081354548 task status = finished
>2012-12-17 09:21:23,548 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling ended, spm status: SPM
>2012-12-17 09:21:23,550 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=25511a55-284e-406f-ae19-8ed081354548), log id: 7feacef9
>2012-12-17 09:21:23,568 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, HSMClearTaskVDSCommand, log id: 7feacef9
>2012-12-17 09:21:23,568 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@10ba7f0b, log id: 7651981b
>2012-12-17 09:21:23,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:21:23,604 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:23,606 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7bfe10e6
>2012-12-17 09:21:23,626 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7bfe10e6
>2012-12-17 09:21:23,626 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: aa4a356
>2012-12-17 09:21:23,626 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1240becc] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:21:23,626 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6507aba2
>2012-12-17 09:21:23,626 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:23,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4a112126
>2012-12-17 09:21:23,646 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4a112126
>2012-12-17 09:21:23,646 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6507aba2
>2012-12-17 09:21:23,646 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1240becc] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:21:24,894 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:21:32,489 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [746c8fa6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:21:32,705 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [13d7977] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:21:32,783 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5d64de4f
>2012-12-17 09:21:32,798 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4a6a04b5
>2012-12-17 09:21:32,818 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [13d7977] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:32,952 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, SpmStopVDSCommand, log id: 4a6a04b5
>2012-12-17 09:21:32,965 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, ResetIrsVDSCommand, log id: 5d64de4f
>2012-12-17 09:21:32,988 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c7a0cb0
>2012-12-17 09:21:32,988 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [13d7977] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:32,988 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c7a0cb0
>2012-12-17 09:21:35,094 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2a365c8d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:21:43,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [2dc4111d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:21:43,141 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [2dc4111d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4829
>2012-12-17 09:21:43,143 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4829, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2e7895dd
>2012-12-17 09:21:43,170 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] spmStart polling started: taskId = 8e733400-8413-40ae-a18d-3c99f65dc527
>2012-12-17 09:21:44,195 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] spmStart polling ended: taskId = 8e733400-8413-40ae-a18d-3c99f65dc527 task status = finished
>2012-12-17 09:21:44,214 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] spmStart polling ended, spm status: SPM
>2012-12-17 09:21:44,215 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8e733400-8413-40ae-a18d-3c99f65dc527), log id: 29233a1f
>2012-12-17 09:21:44,233 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] FINISH, HSMClearTaskVDSCommand, log id: 29233a1f
>2012-12-17 09:21:44,233 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@47bcde52, log id: 2e7895dd
>2012-12-17 09:21:44,245 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [2dc4111d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:21:44,303 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:21:44,303 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 479454a
>2012-12-17 09:21:44,343 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [2dc4111d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:21:44,488 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [1ed39251] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:21:44,563 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [1ed39251] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 68693dad
>2012-12-17 09:21:44,564 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [1ed39251] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 25a81e0b
>2012-12-17 09:21:44,584 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [1ed39251] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:44,817 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [1ed39251] FINISH, SpmStopVDSCommand, log id: 25a81e0b
>2012-12-17 09:21:44,831 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [1ed39251] FINISH, ResetIrsVDSCommand, log id: 68693dad
>2012-12-17 09:21:44,842 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [1ed39251] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4046b736
>2012-12-17 09:21:44,842 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1ed39251] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:44,842 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [1ed39251] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4046b736
>2012-12-17 09:21:44,945 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:21:44,973 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4830
>2012-12-17 09:21:44,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4830, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: da0e83d
>2012-12-17 09:21:44,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling started: taskId = f279b8bc-cb3d-40cc-b0b8-bcc2d22e5c07
>2012-12-17 09:21:46,016 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended: taskId = f279b8bc-cb3d-40cc-b0b8-bcc2d22e5c07 task status = finished
>2012-12-17 09:21:46,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended, spm status: SPM
>2012-12-17 09:21:46,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f279b8bc-cb3d-40cc-b0b8-bcc2d22e5c07), log id: 4f379825
>2012-12-17 09:21:46,053 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMClearTaskVDSCommand, log id: 4f379825
>2012-12-17 09:21:46,053 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4ae5cdc4, log id: da0e83d
>2012-12-17 09:21:46,422 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:21:46,535 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:46,540 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d07578
>2012-12-17 09:21:46,558 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d07578
>2012-12-17 09:21:46,558 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 479454a
>2012-12-17 09:21:46,558 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:21:46,559 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 38d01c43
>2012-12-17 09:21:46,559 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:46,560 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 24a9b363
>2012-12-17 09:21:46,578 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 24a9b363
>2012-12-17 09:21:46,578 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 38d01c43
>2012-12-17 09:21:46,578 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:21:55,011 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [757df811] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:21:55,134 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [160c510e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:21:55,176 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5ca2688e
>2012-12-17 09:21:55,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3b8717c1
>2012-12-17 09:21:55,198 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [160c510e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:55,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, SpmStopVDSCommand, log id: 3b8717c1
>2012-12-17 09:21:55,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, ResetIrsVDSCommand, log id: 5ca2688e
>2012-12-17 09:21:55,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 725f5a0f
>2012-12-17 09:21:55,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:21:55,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 725f5a0f
>2012-12-17 09:21:56,495 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [40c8f45e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:22:05,576 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:22:06,405 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4831
>2012-12-17 09:22:06,407 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4831, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 286c40fa
>2012-12-17 09:22:06,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] spmStart polling started: taskId = 07f607b4-8398-4ddf-af11-27aba8b9e7bc
>2012-12-17 09:22:07,451 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] spmStart polling ended: taskId = 07f607b4-8398-4ddf-af11-27aba8b9e7bc task status = finished
>2012-12-17 09:22:07,470 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] spmStart polling ended, spm status: SPM
>2012-12-17 09:22:07,472 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=07f607b4-8398-4ddf-af11-27aba8b9e7bc), log id: 5023a60c
>2012-12-17 09:22:07,490 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] FINISH, HSMClearTaskVDSCommand, log id: 5023a60c
>2012-12-17 09:22:07,490 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@546a86ee, log id: 286c40fa
>2012-12-17 09:22:07,498 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:22:07,539 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [3299bae4] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:22:07,540 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1f6a4983
>2012-12-17 09:22:07,572 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:22:07,714 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [29b31be1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:22:07,769 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 20b9c0f3
>2012-12-17 09:22:07,770 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 604a0eb1
>2012-12-17 09:22:07,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:08,091 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, SpmStopVDSCommand, log id: 604a0eb1
>2012-12-17 09:22:08,112 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, ResetIrsVDSCommand, log id: 20b9c0f3
>2012-12-17 09:22:08,123 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 50a4f8c9
>2012-12-17 09:22:08,123 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [29b31be1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:08,123 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 50a4f8c9
>2012-12-17 09:22:08,190 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:22:08,203 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4832
>2012-12-17 09:22:08,204 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4832, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 59ad3443
>2012-12-17 09:22:08,226 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] spmStart polling started: taskId = 6b5141df-3a54-4ceb-a464-b18fffd8753a
>2012-12-17 09:22:09,246 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] spmStart polling ended: taskId = 6b5141df-3a54-4ceb-a464-b18fffd8753a task status = finished
>2012-12-17 09:22:09,264 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] spmStart polling ended, spm status: SPM
>2012-12-17 09:22:09,265 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6b5141df-3a54-4ceb-a464-b18fffd8753a), log id: 20441ed0
>2012-12-17 09:22:09,284 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, HSMClearTaskVDSCommand, log id: 20441ed0
>2012-12-17 09:22:09,284 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5842fa3c, log id: 59ad3443
>2012-12-17 09:22:09,306 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:22:09,326 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:09,331 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 442f0660
>2012-12-17 09:22:09,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 442f0660
>2012-12-17 09:22:09,349 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1f6a4983
>2012-12-17 09:22:09,349 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [3299bae4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:22:09,349 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6b4bab98
>2012-12-17 09:22:09,349 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:09,351 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4bcb3a
>2012-12-17 09:22:09,369 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4bcb3a
>2012-12-17 09:22:09,369 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6b4bab98
>2012-12-17 09:22:09,369 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [3299bae4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:22:17,174 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:22:18,236 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [25c7cf42] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:22:18,556 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [5d07077e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:22:18,655 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3a75e4d7
>2012-12-17 09:22:18,656 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7ed0c979
>2012-12-17 09:22:18,676 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:18,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] FINISH, SpmStopVDSCommand, log id: 7ed0c979
>2012-12-17 09:22:18,826 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] FINISH, ResetIrsVDSCommand, log id: 3a75e4d7
>2012-12-17 09:22:18,837 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 74fb38c7
>2012-12-17 09:22:18,837 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5d07077e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:18,837 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 74fb38c7
>2012-12-17 09:22:27,350 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4f973759] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:22:29,061 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [1513ca5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:22:29,075 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [1513ca5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4833
>2012-12-17 09:22:29,077 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4833, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 656f51fe
>2012-12-17 09:22:29,100 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] spmStart polling started: taskId = c2b83930-508a-4770-88da-6a8e6a4dc82b
>2012-12-17 09:22:30,121 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] spmStart polling ended: taskId = c2b83930-508a-4770-88da-6a8e6a4dc82b task status = finished
>2012-12-17 09:22:30,140 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] spmStart polling ended, spm status: SPM
>2012-12-17 09:22:30,141 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c2b83930-508a-4770-88da-6a8e6a4dc82b), log id: 60009592
>2012-12-17 09:22:30,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] FINISH, HSMClearTaskVDSCommand, log id: 60009592
>2012-12-17 09:22:30,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@465bd57b, log id: 656f51fe
>2012-12-17 09:22:30,182 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [1513ca5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:22:30,241 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [779086b5] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:22:30,241 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 59d58215
>2012-12-17 09:22:30,278 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [1513ca5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:22:31,629 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [26d9e89a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:22:31,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 507a42a2
>2012-12-17 09:22:31,746 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 55ad106c
>2012-12-17 09:22:31,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:31,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SpmStopVDSCommand, log id: 55ad106c
>2012-12-17 09:22:31,947 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, ResetIrsVDSCommand, log id: 507a42a2
>2012-12-17 09:22:31,958 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 79486898
>2012-12-17 09:22:31,958 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:31,958 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 79486898
>2012-12-17 09:22:32,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:22:32,120 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4834
>2012-12-17 09:22:32,121 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4834, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 259060ea
>2012-12-17 09:22:32,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] spmStart polling started: taskId = 63805827-a23d-46f9-aacf-fa9b638a4fde
>2012-12-17 09:22:33,172 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] spmStart polling ended: taskId = 63805827-a23d-46f9-aacf-fa9b638a4fde task status = finished
>2012-12-17 09:22:33,191 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] spmStart polling ended, spm status: SPM
>2012-12-17 09:22:33,192 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=63805827-a23d-46f9-aacf-fa9b638a4fde), log id: 2c80b98e
>2012-12-17 09:22:33,211 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, HSMClearTaskVDSCommand, log id: 2c80b98e
>2012-12-17 09:22:33,211 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@37163075, log id: 259060ea
>2012-12-17 09:22:33,219 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:22:33,236 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:33,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 48b366c1
>2012-12-17 09:22:33,257 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 48b366c1
>2012-12-17 09:22:33,257 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 59d58215
>2012-12-17 09:22:33,257 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [779086b5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:22:33,258 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 15118ff3
>2012-12-17 09:22:33,258 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:33,259 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 629c5c2a
>2012-12-17 09:22:33,277 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 629c5c2a
>2012-12-17 09:22:33,277 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 15118ff3
>2012-12-17 09:22:33,277 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [779086b5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:22:37,725 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:22:42,096 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:22:42,240 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [15f75f8d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:22:42,377 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [15f75f8d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 23377118
>2012-12-17 09:22:42,379 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [15f75f8d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e4ce48
>2012-12-17 09:22:42,398 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [15f75f8d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:43,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [15f75f8d] FINISH, SpmStopVDSCommand, log id: 7e4ce48
>2012-12-17 09:22:43,238 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [15f75f8d] FINISH, ResetIrsVDSCommand, log id: 23377118
>2012-12-17 09:22:43,249 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [15f75f8d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6394bb05
>2012-12-17 09:22:43,249 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [15f75f8d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:43,249 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [15f75f8d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6394bb05
>2012-12-17 09:22:47,905 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:22:53,324 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:22:53,334 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4835
>2012-12-17 09:22:53,335 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4835, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 799ad51d
>2012-12-17 09:22:53,356 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] spmStart polling started: taskId = 2fe3b427-5fa5-4326-9b21-ba1054903ea1
>2012-12-17 09:22:54,380 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] spmStart polling ended: taskId = 2fe3b427-5fa5-4326-9b21-ba1054903ea1 task status = finished
>2012-12-17 09:22:54,398 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] spmStart polling ended, spm status: SPM
>2012-12-17 09:22:54,399 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2fe3b427-5fa5-4326-9b21-ba1054903ea1), log id: 37da5da4
>2012-12-17 09:22:54,417 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, HSMClearTaskVDSCommand, log id: 37da5da4
>2012-12-17 09:22:54,417 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6139d6a1, log id: 799ad51d
>2012-12-17 09:22:54,441 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:22:54,475 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-9) [29b31be1] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:22:54,476 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 44211788
>2012-12-17 09:22:54,517 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:22:54,686 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-62) [4f3078ed] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:22:54,855 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2bed9894
>2012-12-17 09:22:54,857 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6c277eb0
>2012-12-17 09:22:54,879 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:55,082 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, SpmStopVDSCommand, log id: 6c277eb0
>2012-12-17 09:22:55,094 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, ResetIrsVDSCommand, log id: 2bed9894
>2012-12-17 09:22:55,104 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b45e087
>2012-12-17 09:22:55,105 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:55,105 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b45e087
>2012-12-17 09:22:55,179 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [29b31be1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:22:55,207 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [29b31be1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4836
>2012-12-17 09:22:55,209 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4836, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 76474bae
>2012-12-17 09:22:55,231 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] spmStart polling started: taskId = 24c8b5cf-a9ff-4c3c-a25b-40d9bd9593cd
>2012-12-17 09:22:56,251 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] spmStart polling ended: taskId = 24c8b5cf-a9ff-4c3c-a25b-40d9bd9593cd task status = finished
>2012-12-17 09:22:56,270 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] spmStart polling ended, spm status: SPM
>2012-12-17 09:22:56,271 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=24c8b5cf-a9ff-4c3c-a25b-40d9bd9593cd), log id: 2761c1c
>2012-12-17 09:22:56,290 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, HSMClearTaskVDSCommand, log id: 2761c1c
>2012-12-17 09:22:56,290 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@182f7387, log id: 76474bae
>2012-12-17 09:22:56,302 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [29b31be1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:22:56,317 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:56,319 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 685b56dd
>2012-12-17 09:22:56,337 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 685b56dd
>2012-12-17 09:22:56,337 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 44211788
>2012-12-17 09:22:56,337 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-9) [29b31be1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:22:56,338 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 44a3fcb1
>2012-12-17 09:22:56,338 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:22:56,339 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1c79b091
>2012-12-17 09:22:56,357 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1c79b091
>2012-12-17 09:22:56,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 44a3fcb1
>2012-12-17 09:22:56,357 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-9) [29b31be1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:22:58,120 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:23:05,284 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [11c9636b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:23:05,521 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [5389729] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:23:06,147 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 38c7f479
>2012-12-17 09:23:06,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 25aad06f
>2012-12-17 09:23:06,169 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [5389729] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:23:06,327 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, SpmStopVDSCommand, log id: 25aad06f
>2012-12-17 09:23:06,356 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, ResetIrsVDSCommand, log id: 38c7f479
>2012-12-17 09:23:06,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: d60f920
>2012-12-17 09:23:06,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [5389729] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:23:06,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, MarkPoolInReconstructModeVDSCommand, log id: d60f920
>2012-12-17 09:23:08,310 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:23:16,417 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:23:16,451 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4837
>2012-12-17 09:23:16,452 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4837, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4a4ad157
>2012-12-17 09:23:16,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] spmStart polling started: taskId = 19c20a09-2566-4482-9e7b-bdb4ad0d84db
>2012-12-17 09:23:17,498 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] spmStart polling ended: taskId = 19c20a09-2566-4482-9e7b-bdb4ad0d84db task status = finished
>2012-12-17 09:23:17,516 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] spmStart polling ended, spm status: SPM
>2012-12-17 09:23:17,517 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=19c20a09-2566-4482-9e7b-bdb4ad0d84db), log id: 687b0caa
>2012-12-17 09:23:17,536 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, HSMClearTaskVDSCommand, log id: 687b0caa
>2012-12-17 09:23:17,536 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@667ca27c, log id: 4a4ad157
>2012-12-17 09:23:17,549 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:23:17,648 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-29) [57cdd675] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:23:17,648 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3118d901
>2012-12-17 09:23:17,665 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:23:17,799 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-45) [10684b21] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:23:17,883 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [10684b21] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7e73b609
>2012-12-17 09:23:17,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [10684b21] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2f4ff669
>2012-12-17 09:23:17,905 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [10684b21] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:23:18,036 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [10684b21] FINISH, SpmStopVDSCommand, log id: 2f4ff669
>2012-12-17 09:23:18,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [10684b21] FINISH, ResetIrsVDSCommand, log id: 7e73b609
>2012-12-17 09:23:18,062 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [10684b21] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: c519eb8
>2012-12-17 09:23:18,062 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [10684b21] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:23:18,062 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [10684b21] FINISH, MarkPoolInReconstructModeVDSCommand, log id: c519eb8
>2012-12-17 09:23:18,113 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:23:18,131 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4838
>2012-12-17 09:23:18,132 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4838, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40a9aa9b
>2012-12-17 09:23:18,154 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] spmStart polling started: taskId = c9a10d00-e1e1-4748-a6d8-a5e551beae5b
>2012-12-17 09:23:19,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] spmStart polling ended: taskId = c9a10d00-e1e1-4748-a6d8-a5e551beae5b task status = finished
>2012-12-17 09:23:19,192 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] spmStart polling ended, spm status: SPM
>2012-12-17 09:23:19,194 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c9a10d00-e1e1-4748-a6d8-a5e551beae5b), log id: 5fd28ca5
>2012-12-17 09:23:19,212 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, HSMClearTaskVDSCommand, log id: 5fd28ca5
>2012-12-17 09:23:19,212 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@780fd5c8, log id: 40a9aa9b
>2012-12-17 09:23:19,239 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:23:19,255 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:23:19,259 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 30ba2e28
>2012-12-17 09:23:19,279 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 30ba2e28
>2012-12-17 09:23:19,279 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3118d901
>2012-12-17 09:23:19,279 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-29) [57cdd675] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:23:19,280 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: f2c5d16
>2012-12-17 09:23:19,280 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:23:19,282 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3f357b97
>2012-12-17 09:23:19,300 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3f357b97
>2012-12-17 09:23:19,300 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: f2c5d16
>2012-12-17 09:23:19,300 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-29) [57cdd675] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:23:28,150 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:23:28,272 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [492c9849] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:23:28,336 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5e30530d
>2012-12-17 09:23:28,338 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2f679194
>2012-12-17 09:23:28,358 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [492c9849] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:23:28,804 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [1cfebcc3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:23:28,949 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, SpmStopVDSCommand, log id: 2f679194
>2012-12-17 09:23:28,985 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, ResetIrsVDSCommand, log id: 5e30530d
>2012-12-17 09:23:28,996 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 34e729dc
>2012-12-17 09:23:28,996 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:23:28,996 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 34e729dc
>2012-12-17 09:23:39,066 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:23:39,079 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4839
>2012-12-17 09:23:39,081 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4839, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4597a118
>2012-12-17 09:23:39,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] spmStart polling started: taskId = cf1fcb5a-1d7e-45cc-9b0e-98f882d2ef83
>2012-12-17 09:23:40,126 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] spmStart polling ended: taskId = cf1fcb5a-1d7e-45cc-9b0e-98f882d2ef83 task status = finished
>2012-12-17 09:23:40,145 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] spmStart polling ended, spm status: SPM
>2012-12-17 09:23:40,147 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cf1fcb5a-1d7e-45cc-9b0e-98f882d2ef83), log id: 603f2b95
>2012-12-17 09:23:40,166 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, HSMClearTaskVDSCommand, log id: 603f2b95
>2012-12-17 09:23:40,166 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@53467490, log id: 4597a118
>2012-12-17 09:23:40,186 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:23:40,248 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:23:40,248 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 68005028
>2012-12-17 09:23:40,257 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:23:41,019 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [8d9c42c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:23:41,199 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [8d9c42c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 64418f3
>2012-12-17 09:23:41,201 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [8d9c42c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: d83a734
>2012-12-17 09:23:41,223 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [8d9c42c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:23:41,713 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [8d9c42c] FINISH, SpmStopVDSCommand, log id: d83a734
>2012-12-17 09:23:41,730 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [8d9c42c] FINISH, ResetIrsVDSCommand, log id: 64418f3
>2012-12-17 09:23:41,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [8d9c42c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6a37099d
>2012-12-17 09:23:41,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [8d9c42c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:23:41,743 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [8d9c42c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6a37099d
>2012-12-17 09:23:41,842 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:23:41,866 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4840
>2012-12-17 09:23:41,867 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4840, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7410820
>2012-12-17 09:23:41,889 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling started: taskId = b3c0c5c0-85b1-40b8-b4f1-b83ba5afe9d8
>2012-12-17 09:23:42,911 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling ended: taskId = b3c0c5c0-85b1-40b8-b4f1-b83ba5afe9d8 task status = finished
>2012-12-17 09:23:42,929 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling ended, spm status: SPM
>2012-12-17 09:23:42,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b3c0c5c0-85b1-40b8-b4f1-b83ba5afe9d8), log id: 74f49533
>2012-12-17 09:23:42,948 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMClearTaskVDSCommand, log id: 74f49533
>2012-12-17 09:23:42,948 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@656f8341, log id: 7410820
>2012-12-17 09:23:42,961 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:23:42,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:23:42,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7a6ba0bf
>2012-12-17 09:23:43,016 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7a6ba0bf
>2012-12-17 09:23:43,016 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 68005028
>2012-12-17 09:23:43,016 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:23:43,016 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 31eb6dce
>2012-12-17 09:23:43,016 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:23:43,018 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 21316151
>2012-12-17 09:23:43,043 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 21316151
>2012-12-17 09:23:43,043 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 31eb6dce
>2012-12-17 09:23:43,043 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:23:49,235 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:23:51,908 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [1091267a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:23:52,104 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:23:52,169 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 73d89590
>2012-12-17 09:23:52,170 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4b4f0d60
>2012-12-17 09:23:52,190 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:23:52,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] FINISH, SpmStopVDSCommand, log id: 4b4f0d60
>2012-12-17 09:23:52,388 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] FINISH, ResetIrsVDSCommand, log id: 73d89590
>2012-12-17 09:23:52,403 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44746e7e
>2012-12-17 09:23:52,403 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:23:52,403 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44746e7e
>2012-12-17 09:23:59,549 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2a365c8d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:24:02,527 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4f973759] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:24:02,557 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4f973759] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4841
>2012-12-17 09:24:02,558 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [4f973759] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4841, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6c45670
>2012-12-17 09:24:02,583 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [4f973759] spmStart polling started: taskId = 965be689-3e54-4f45-aed3-ede1fafe8c34
>2012-12-17 09:24:03,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [4f973759] spmStart polling ended: taskId = 965be689-3e54-4f45-aed3-ede1fafe8c34 task status = finished
>2012-12-17 09:24:03,639 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [4f973759] spmStart polling ended, spm status: SPM
>2012-12-17 09:24:03,640 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [4f973759] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=965be689-3e54-4f45-aed3-ede1fafe8c34), log id: 317cf53a
>2012-12-17 09:24:03,659 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [4f973759] FINISH, HSMClearTaskVDSCommand, log id: 317cf53a
>2012-12-17 09:24:03,659 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [4f973759] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2c7cc38, log id: 6c45670
>2012-12-17 09:24:03,705 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4f973759] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:24:03,800 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:24:03,800 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 503c8411
>2012-12-17 09:24:03,839 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4f973759] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:24:03,974 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [2bb64a93] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:24:04,083 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [2bb64a93] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 10aaa9d8
>2012-12-17 09:24:04,084 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [2bb64a93] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 78acda63
>2012-12-17 09:24:04,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [2bb64a93] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:04,378 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [2bb64a93] FINISH, SpmStopVDSCommand, log id: 78acda63
>2012-12-17 09:24:04,399 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [2bb64a93] FINISH, ResetIrsVDSCommand, log id: 10aaa9d8
>2012-12-17 09:24:04,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [2bb64a93] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 57f60efd
>2012-12-17 09:24:04,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [2bb64a93] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:04,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [2bb64a93] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 57f60efd
>2012-12-17 09:24:04,498 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:24:04,510 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4842
>2012-12-17 09:24:04,511 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4842, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4b2a2d4c
>2012-12-17 09:24:04,533 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling started: taskId = ac189118-8df0-4262-aefe-ff216783a77a
>2012-12-17 09:24:05,556 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended: taskId = ac189118-8df0-4262-aefe-ff216783a77a task status = finished
>2012-12-17 09:24:05,575 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended, spm status: SPM
>2012-12-17 09:24:05,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ac189118-8df0-4262-aefe-ff216783a77a), log id: 3d51b29c
>2012-12-17 09:24:05,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMClearTaskVDSCommand, log id: 3d51b29c
>2012-12-17 09:24:05,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3ea53f37, log id: 4b2a2d4c
>2012-12-17 09:24:06,173 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:24:06,218 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:06,224 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7d5a967
>2012-12-17 09:24:06,245 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7d5a967
>2012-12-17 09:24:06,245 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 503c8411
>2012-12-17 09:24:06,245 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:24:06,245 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4f4f36ee
>2012-12-17 09:24:06,245 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:06,246 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4d9b2939
>2012-12-17 09:24:06,265 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4d9b2939
>2012-12-17 09:24:06,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4f4f36ee
>2012-12-17 09:24:06,265 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:24:09,783 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:24:14,550 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [14ba2ea4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:24:14,694 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [127bfc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:24:14,821 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6c0fe961
>2012-12-17 09:24:14,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 251a3248
>2012-12-17 09:24:14,842 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [127bfc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:15,008 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, SpmStopVDSCommand, log id: 251a3248
>2012-12-17 09:24:15,039 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, ResetIrsVDSCommand, log id: 6c0fe961
>2012-12-17 09:24:15,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2d573d2d
>2012-12-17 09:24:15,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:15,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2d573d2d
>2012-12-17 09:24:20,014 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7e80f448] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:24:25,153 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:24:25,211 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4843
>2012-12-17 09:24:25,212 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4843, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6af5d471
>2012-12-17 09:24:25,235 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] spmStart polling started: taskId = a3375965-d815-4a08-abd7-6b140099c88f
>2012-12-17 09:24:26,257 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] spmStart polling ended: taskId = a3375965-d815-4a08-abd7-6b140099c88f task status = finished
>2012-12-17 09:24:26,275 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] spmStart polling ended, spm status: SPM
>2012-12-17 09:24:26,276 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a3375965-d815-4a08-abd7-6b140099c88f), log id: 71a735df
>2012-12-17 09:24:26,295 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, HSMClearTaskVDSCommand, log id: 71a735df
>2012-12-17 09:24:26,295 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@18c1fdc4, log id: 6af5d471
>2012-12-17 09:24:26,573 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:24:26,827 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [6c899ac8] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:24:26,827 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5457e27a
>2012-12-17 09:24:26,857 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:24:27,066 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [6b59f114] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:24:27,112 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5852e09d
>2012-12-17 09:24:27,113 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 170a08a4
>2012-12-17 09:24:27,133 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:27,275 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] FINISH, SpmStopVDSCommand, log id: 170a08a4
>2012-12-17 09:24:27,289 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] FINISH, ResetIrsVDSCommand, log id: 5852e09d
>2012-12-17 09:24:27,300 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2f6ca323
>2012-12-17 09:24:27,300 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6b59f114] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:27,300 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2f6ca323
>2012-12-17 09:24:27,351 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:24:27,363 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4844
>2012-12-17 09:24:27,364 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4844, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 375803e1
>2012-12-17 09:24:27,388 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] spmStart polling started: taskId = 7c4d6ffd-9b5d-473c-b660-d69564859127
>2012-12-17 09:24:28,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] spmStart polling ended: taskId = 7c4d6ffd-9b5d-473c-b660-d69564859127 task status = finished
>2012-12-17 09:24:28,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] spmStart polling ended, spm status: SPM
>2012-12-17 09:24:28,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7c4d6ffd-9b5d-473c-b660-d69564859127), log id: 70f48923
>2012-12-17 09:24:28,448 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, HSMClearTaskVDSCommand, log id: 70f48923
>2012-12-17 09:24:28,448 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@465792c4, log id: 375803e1
>2012-12-17 09:24:28,483 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:24:28,515 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:28,520 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 9b9f510
>2012-12-17 09:24:28,539 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 9b9f510
>2012-12-17 09:24:28,539 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5457e27a
>2012-12-17 09:24:28,539 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [6c899ac8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:24:28,539 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: f87f26d
>2012-12-17 09:24:28,539 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:28,540 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 49ac91b
>2012-12-17 09:24:28,559 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 49ac91b
>2012-12-17 09:24:28,559 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: f87f26d
>2012-12-17 09:24:28,560 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [6c899ac8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:24:30,266 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [efbb817] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:24:37,416 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [50bce313] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:24:37,610 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [78e614fb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:24:37,677 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [78e614fb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6520d334
>2012-12-17 09:24:37,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [78e614fb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2edbe137
>2012-12-17 09:24:37,698 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [78e614fb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:37,838 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [78e614fb] FINISH, SpmStopVDSCommand, log id: 2edbe137
>2012-12-17 09:24:37,850 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [78e614fb] FINISH, ResetIrsVDSCommand, log id: 6520d334
>2012-12-17 09:24:37,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [78e614fb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 72065448
>2012-12-17 09:24:37,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [78e614fb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:37,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [78e614fb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 72065448
>2012-12-17 09:24:41,640 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:24:47,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [356a6c1f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:24:47,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [356a6c1f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4845
>2012-12-17 09:24:47,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4845, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7868539b
>2012-12-17 09:24:47,954 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] spmStart polling started: taskId = 3bea3675-b1b2-435c-b66b-848e8cfcbde2
>2012-12-17 09:24:48,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] spmStart polling ended: taskId = 3bea3675-b1b2-435c-b66b-848e8cfcbde2 task status = finished
>2012-12-17 09:24:48,994 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] spmStart polling ended, spm status: SPM
>2012-12-17 09:24:48,995 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3bea3675-b1b2-435c-b66b-848e8cfcbde2), log id: 7015ce46
>2012-12-17 09:24:49,014 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] FINISH, HSMClearTaskVDSCommand, log id: 7015ce46
>2012-12-17 09:24:49,014 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@27998f27, log id: 7868539b
>2012-12-17 09:24:49,025 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [356a6c1f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:24:49,075 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:24:49,076 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1e52ba00
>2012-12-17 09:24:49,101 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [356a6c1f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:24:49,290 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [3e7d3817] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:24:49,359 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 13636d9a
>2012-12-17 09:24:49,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3edd8e16
>2012-12-17 09:24:49,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:49,521 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SpmStopVDSCommand, log id: 3edd8e16
>2012-12-17 09:24:49,535 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, ResetIrsVDSCommand, log id: 13636d9a
>2012-12-17 09:24:49,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 11d625e0
>2012-12-17 09:24:49,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:49,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 11d625e0
>2012-12-17 09:24:49,590 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:24:49,618 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4846
>2012-12-17 09:24:49,620 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4846, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2bf83726
>2012-12-17 09:24:49,641 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling started: taskId = 0f536468-8aea-4ef0-ad0c-49717ee13738
>2012-12-17 09:24:50,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling ended: taskId = 0f536468-8aea-4ef0-ad0c-49717ee13738 task status = finished
>2012-12-17 09:24:50,682 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling ended, spm status: SPM
>2012-12-17 09:24:50,683 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0f536468-8aea-4ef0-ad0c-49717ee13738), log id: 40248a40
>2012-12-17 09:24:50,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMClearTaskVDSCommand, log id: 40248a40
>2012-12-17 09:24:50,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1ef090a6, log id: 2bf83726
>2012-12-17 09:24:51,018 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:24:51,248 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:51,254 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 17b1caa3
>2012-12-17 09:24:51,276 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 17b1caa3
>2012-12-17 09:24:51,276 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1e52ba00
>2012-12-17 09:24:51,276 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:24:51,277 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 41f61374
>2012-12-17 09:24:51,277 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:24:51,278 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 47f95728
>2012-12-17 09:24:51,297 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 47f95728
>2012-12-17 09:24:51,297 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 41f61374
>2012-12-17 09:24:51,298 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:24:51,861 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [2bb64a93] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:24:59,691 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2ed19e27] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:24:59,834 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [3c6804c8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:24:59,895 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 295a0ab5
>2012-12-17 09:24:59,896 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 10c268a6
>2012-12-17 09:24:59,916 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [61136831] Checking autorecoverable hosts
>2012-12-17 09:25:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [61136831] Autorecovering 0 hosts
>2012-12-17 09:25:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [61136831] Checking autorecoverable hosts done
>2012-12-17 09:25:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [61136831] Autorecovering storage domains is disabled, skipping
>2012-12-17 09:25:00,062 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] FINISH, SpmStopVDSCommand, log id: 10c268a6
>2012-12-17 09:25:00,078 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] FINISH, ResetIrsVDSCommand, log id: 295a0ab5
>2012-12-17 09:25:00,089 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b4970d4
>2012-12-17 09:25:00,089 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3c6804c8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:00,089 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b4970d4
>2012-12-17 09:25:02,124 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [1cfebcc3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:25:10,162 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:25:10,182 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4847
>2012-12-17 09:25:10,183 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4847, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 8a1114d
>2012-12-17 09:25:10,205 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] spmStart polling started: taskId = 83f45c58-a8a4-4ea5-b2e2-c236d58d1b28
>2012-12-17 09:25:11,229 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] spmStart polling ended: taskId = 83f45c58-a8a4-4ea5-b2e2-c236d58d1b28 task status = finished
>2012-12-17 09:25:11,247 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] spmStart polling ended, spm status: SPM
>2012-12-17 09:25:11,249 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=83f45c58-a8a4-4ea5-b2e2-c236d58d1b28), log id: 2b25f467
>2012-12-17 09:25:11,267 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, HSMClearTaskVDSCommand, log id: 2b25f467
>2012-12-17 09:25:11,267 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48d0ac52, log id: 8a1114d
>2012-12-17 09:25:11,421 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:25:11,663 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [3a1c2706] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:25:11,663 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3644438c
>2012-12-17 09:25:11,748 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:25:12,042 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-68) [78864454] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:25:12,121 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [78864454] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 61f23c49
>2012-12-17 09:25:12,123 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [78864454] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 47bd7efc
>2012-12-17 09:25:12,144 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [78864454] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:12,268 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [78864454] FINISH, SpmStopVDSCommand, log id: 47bd7efc
>2012-12-17 09:25:12,302 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [78864454] FINISH, ResetIrsVDSCommand, log id: 61f23c49
>2012-12-17 09:25:12,314 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [78864454] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c6ef5a7
>2012-12-17 09:25:12,314 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [78864454] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:12,314 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [78864454] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c6ef5a7
>2012-12-17 09:25:12,405 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:25:12,421 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [570659e5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:25:12,489 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4848
>2012-12-17 09:25:12,491 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4848, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3ad66dd3
>2012-12-17 09:25:12,515 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] spmStart polling started: taskId = 9cbcf23f-acc2-43ac-80d3-ae62ba945a39
>2012-12-17 09:25:13,537 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] spmStart polling ended: taskId = 9cbcf23f-acc2-43ac-80d3-ae62ba945a39 task status = finished
>2012-12-17 09:25:13,556 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] spmStart polling ended, spm status: SPM
>2012-12-17 09:25:13,557 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9cbcf23f-acc2-43ac-80d3-ae62ba945a39), log id: 7fb909f4
>2012-12-17 09:25:13,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, HSMClearTaskVDSCommand, log id: 7fb909f4
>2012-12-17 09:25:13,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4d3cb3ae, log id: 3ad66dd3
>2012-12-17 09:25:13,594 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:25:13,648 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:13,652 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 29382d2e
>2012-12-17 09:25:13,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 29382d2e
>2012-12-17 09:25:13,674 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3644438c
>2012-12-17 09:25:13,674 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [3a1c2706] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:25:13,675 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 271e3c26
>2012-12-17 09:25:13,675 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:13,676 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 684c0f1f
>2012-12-17 09:25:13,696 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 684c0f1f
>2012-12-17 09:25:13,697 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 271e3c26
>2012-12-17 09:25:13,697 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [3a1c2706] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:25:13,738 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [69834199] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:25:13,738 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3b130c3a
>2012-12-17 09:25:13,739 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:13,741 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: d9645da
>2012-12-17 09:25:13,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: d9645da
>2012-12-17 09:25:13,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3b130c3a
>2012-12-17 09:25:13,765 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [69834199] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:25:13,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 12f7cfdc
>2012-12-17 09:25:13,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:13,767 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4beaf561
>2012-12-17 09:25:13,786 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4beaf561
>2012-12-17 09:25:13,786 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 12f7cfdc
>2012-12-17 09:25:13,786 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [69834199] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:25:22,506 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [7f36fad2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:25:22,828 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [550a1bf7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:25:22,928 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 47912c94
>2012-12-17 09:25:22,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 54847281
>2012-12-17 09:25:22,950 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:23,672 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] FINISH, SpmStopVDSCommand, log id: 54847281
>2012-12-17 09:25:23,702 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] FINISH, ResetIrsVDSCommand, log id: 47912c94
>2012-12-17 09:25:23,712 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2e1a5c65
>2012-12-17 09:25:23,712 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [550a1bf7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:23,712 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2e1a5c65
>2012-12-17 09:25:33,021 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [13d7977] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:25:33,758 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [3eeaa12a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:25:33,789 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [3eeaa12a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4849
>2012-12-17 09:25:33,791 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4849, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 625bcc0b
>2012-12-17 09:25:33,813 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] spmStart polling started: taskId = f0f48372-f3f9-455f-ad5d-c493d5d36334
>2012-12-17 09:25:34,836 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] spmStart polling ended: taskId = f0f48372-f3f9-455f-ad5d-c493d5d36334 task status = finished
>2012-12-17 09:25:34,855 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] spmStart polling ended, spm status: SPM
>2012-12-17 09:25:34,856 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f0f48372-f3f9-455f-ad5d-c493d5d36334), log id: abd074d
>2012-12-17 09:25:34,875 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] FINISH, HSMClearTaskVDSCommand, log id: abd074d
>2012-12-17 09:25:34,875 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2a55bd, log id: 625bcc0b
>2012-12-17 09:25:34,901 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [3eeaa12a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:25:34,972 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [a029edc] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:25:34,972 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4c314c8d
>2012-12-17 09:25:34,984 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [3eeaa12a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:25:35,220 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-97) [36643dd7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:25:35,285 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: dd4f0b2
>2012-12-17 09:25:35,287 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4286f156
>2012-12-17 09:25:35,307 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:35,477 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, SpmStopVDSCommand, log id: 4286f156
>2012-12-17 09:25:36,106 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, ResetIrsVDSCommand, log id: dd4f0b2
>2012-12-17 09:25:36,117 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 59118b5d
>2012-12-17 09:25:36,117 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:36,117 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 59118b5d
>2012-12-17 09:25:36,183 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:25:36,223 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4850
>2012-12-17 09:25:36,224 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4850, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1237ccaf
>2012-12-17 09:25:36,246 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] spmStart polling started: taskId = 750daa0b-3f7e-4c8b-add1-80b61487b11e
>2012-12-17 09:25:37,268 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] spmStart polling ended: taskId = 750daa0b-3f7e-4c8b-add1-80b61487b11e task status = finished
>2012-12-17 09:25:37,286 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] spmStart polling ended, spm status: SPM
>2012-12-17 09:25:37,287 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=750daa0b-3f7e-4c8b-add1-80b61487b11e), log id: 564a58d0
>2012-12-17 09:25:37,306 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, HSMClearTaskVDSCommand, log id: 564a58d0
>2012-12-17 09:25:37,306 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2b0cebf2, log id: 1237ccaf
>2012-12-17 09:25:37,316 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:25:37,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:37,370 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4fbda1de
>2012-12-17 09:25:37,389 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4fbda1de
>2012-12-17 09:25:37,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4c314c8d
>2012-12-17 09:25:37,390 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [a029edc] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:25:37,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2ace1eb8
>2012-12-17 09:25:37,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:37,391 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 58e0114b
>2012-12-17 09:25:37,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 58e0114b
>2012-12-17 09:25:37,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2ace1eb8
>2012-12-17 09:25:37,410 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [a029edc] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:25:43,227 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [327be49e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:25:46,446 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:25:47,003 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [622cf65b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:25:47,108 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [622cf65b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: dd0ecee
>2012-12-17 09:25:47,110 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [622cf65b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4d96372c
>2012-12-17 09:25:47,130 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [622cf65b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:47,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [622cf65b] FINISH, SpmStopVDSCommand, log id: 4d96372c
>2012-12-17 09:25:47,365 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [622cf65b] FINISH, ResetIrsVDSCommand, log id: dd0ecee
>2012-12-17 09:25:47,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [622cf65b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2e8eb342
>2012-12-17 09:25:47,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [622cf65b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:47,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [622cf65b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2e8eb342
>2012-12-17 09:25:51,777 INFO  [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-50) [76c72cb4] DbUserCacheManager::refreshAllUserData() - entered
>2012-12-17 09:25:53,450 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [15f75f8d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:25:57,446 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [6c5669f4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:25:57,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [6c5669f4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4851
>2012-12-17 09:25:57,504 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4851, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4b42984e
>2012-12-17 09:25:57,543 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] spmStart polling started: taskId = d6c93898-85dc-415f-b4a6-d3f58dfcd9e3
>2012-12-17 09:25:58,575 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] spmStart polling ended: taskId = d6c93898-85dc-415f-b4a6-d3f58dfcd9e3 task status = finished
>2012-12-17 09:25:58,593 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] spmStart polling ended, spm status: SPM
>2012-12-17 09:25:58,594 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d6c93898-85dc-415f-b4a6-d3f58dfcd9e3), log id: 1d2ad14e
>2012-12-17 09:25:58,613 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] FINISH, HSMClearTaskVDSCommand, log id: 1d2ad14e
>2012-12-17 09:25:58,613 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@661366d7, log id: 4b42984e
>2012-12-17 09:25:58,632 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [6c5669f4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:25:58,711 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:25:58,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 39edc935
>2012-12-17 09:25:58,743 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [6c5669f4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:25:58,939 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [45dceb6a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:25:59,033 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [45dceb6a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6fea0990
>2012-12-17 09:25:59,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [45dceb6a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4cdd48d9
>2012-12-17 09:25:59,055 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [45dceb6a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:59,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [45dceb6a] FINISH, SpmStopVDSCommand, log id: 4cdd48d9
>2012-12-17 09:25:59,201 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [45dceb6a] FINISH, ResetIrsVDSCommand, log id: 6fea0990
>2012-12-17 09:25:59,212 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [45dceb6a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 638590c3
>2012-12-17 09:25:59,212 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [45dceb6a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:25:59,212 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [45dceb6a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 638590c3
>2012-12-17 09:25:59,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:25:59,278 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4852
>2012-12-17 09:25:59,279 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4852, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40f8571e
>2012-12-17 09:25:59,301 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling started: taskId = fbef59f5-1b2b-4197-82b0-f38ee1590453
>2012-12-17 09:26:00,322 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended: taskId = fbef59f5-1b2b-4197-82b0-f38ee1590453 task status = finished
>2012-12-17 09:26:00,341 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended, spm status: SPM
>2012-12-17 09:26:00,342 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=fbef59f5-1b2b-4197-82b0-f38ee1590453), log id: 3d54a41b
>2012-12-17 09:26:00,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMClearTaskVDSCommand, log id: 3d54a41b
>2012-12-17 09:26:00,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7fe20811, log id: 40f8571e
>2012-12-17 09:26:00,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:26:00,389 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:00,393 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d8c3f01
>2012-12-17 09:26:00,414 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d8c3f01
>2012-12-17 09:26:00,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 39edc935
>2012-12-17 09:26:00,414 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:26:00,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 127c92d7
>2012-12-17 09:26:00,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:00,415 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7f740b86
>2012-12-17 09:26:00,435 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7f740b86
>2012-12-17 09:26:00,435 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 127c92d7
>2012-12-17 09:26:00,435 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:26:03,704 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:26:09,304 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4beddd99] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:26:09,428 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-89) [7bb72f96] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:26:09,542 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [7bb72f96] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 37b2c114
>2012-12-17 09:26:09,544 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [7bb72f96] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7ebbf570
>2012-12-17 09:26:09,565 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [7bb72f96] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:09,738 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [7bb72f96] FINISH, SpmStopVDSCommand, log id: 7ebbf570
>2012-12-17 09:26:09,749 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [7bb72f96] FINISH, ResetIrsVDSCommand, log id: 37b2c114
>2012-12-17 09:26:09,760 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [7bb72f96] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76d493e6
>2012-12-17 09:26:09,760 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [7bb72f96] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:09,760 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [7bb72f96] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76d493e6
>2012-12-17 09:26:13,893 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:26:19,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5ecdfb51] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:26:19,883 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5ecdfb51] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4853
>2012-12-17 09:26:19,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4853, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 613ebde8
>2012-12-17 09:26:19,906 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] spmStart polling started: taskId = 765e2c65-6271-4a40-ac27-e53609941adb
>2012-12-17 09:26:20,929 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] spmStart polling ended: taskId = 765e2c65-6271-4a40-ac27-e53609941adb task status = finished
>2012-12-17 09:26:20,948 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] spmStart polling ended, spm status: SPM
>2012-12-17 09:26:20,949 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=765e2c65-6271-4a40-ac27-e53609941adb), log id: 38d03ac7
>2012-12-17 09:26:20,968 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] FINISH, HSMClearTaskVDSCommand, log id: 38d03ac7
>2012-12-17 09:26:20,968 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6ec2fbf4, log id: 613ebde8
>2012-12-17 09:26:21,134 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5ecdfb51] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:26:21,926 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [3aebafd8] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:26:21,926 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3b7532ab
>2012-12-17 09:26:21,961 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5ecdfb51] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:26:22,125 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [47111e6d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:26:22,186 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 47198520
>2012-12-17 09:26:22,188 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 759ea4d7
>2012-12-17 09:26:22,208 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:23,175 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] FINISH, SpmStopVDSCommand, log id: 759ea4d7
>2012-12-17 09:26:23,200 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] FINISH, ResetIrsVDSCommand, log id: 47198520
>2012-12-17 09:26:23,211 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 8332b77
>2012-12-17 09:26:23,211 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [47111e6d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:23,211 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 8332b77
>2012-12-17 09:26:23,273 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3aebafd8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:26:23,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3aebafd8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4854
>2012-12-17 09:26:23,289 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4854, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 695e79db
>2012-12-17 09:26:23,312 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] spmStart polling started: taskId = 6e015609-4f46-480e-a72e-41e80c606c2c
>2012-12-17 09:26:24,334 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] spmStart polling ended: taskId = 6e015609-4f46-480e-a72e-41e80c606c2c task status = finished
>2012-12-17 09:26:24,358 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] spmStart polling ended, spm status: SPM
>2012-12-17 09:26:24,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6e015609-4f46-480e-a72e-41e80c606c2c), log id: 295ac09d
>2012-12-17 09:26:24,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, HSMClearTaskVDSCommand, log id: 295ac09d
>2012-12-17 09:26:24,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@56c174a5, log id: 695e79db
>2012-12-17 09:26:24,394 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3aebafd8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:26:24,405 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:24,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1968713b
>2012-12-17 09:26:24,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1968713b
>2012-12-17 09:26:24,430 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3b7532ab
>2012-12-17 09:26:24,430 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [3aebafd8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:26:24,430 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 17e1e585
>2012-12-17 09:26:24,430 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:24,431 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 78474385
>2012-12-17 09:26:24,450 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 78474385
>2012-12-17 09:26:24,450 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 17e1e585
>2012-12-17 09:26:24,450 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [3aebafd8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:26:33,331 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [67dedb1b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:26:33,570 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-76) [3fe36f5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:26:33,638 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6f91f287
>2012-12-17 09:26:33,639 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e24560f
>2012-12-17 09:26:33,660 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:34,234 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, SpmStopVDSCommand, log id: 7e24560f
>2012-12-17 09:26:34,276 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, ResetIrsVDSCommand, log id: 6f91f287
>2012-12-17 09:26:34,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7596f185
>2012-12-17 09:26:34,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3fe36f5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:34,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7596f185
>2012-12-17 09:26:34,336 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [622cf65b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:26:44,382 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [48bd7fc3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:26:44,395 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [48bd7fc3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4855
>2012-12-17 09:26:44,397 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4855, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 64234e4d
>2012-12-17 09:26:44,419 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] spmStart polling started: taskId = cb212e4c-b7b4-44f6-bec6-ee0d683bc06c
>2012-12-17 09:26:45,441 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] spmStart polling ended: taskId = cb212e4c-b7b4-44f6-bec6-ee0d683bc06c task status = finished
>2012-12-17 09:26:45,460 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] spmStart polling ended, spm status: SPM
>2012-12-17 09:26:45,461 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cb212e4c-b7b4-44f6-bec6-ee0d683bc06c), log id: 7066ac4f
>2012-12-17 09:26:45,479 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] FINISH, HSMClearTaskVDSCommand, log id: 7066ac4f
>2012-12-17 09:26:45,479 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@64bab02f, log id: 64234e4d
>2012-12-17 09:26:45,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [48bd7fc3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:26:46,461 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [733324dc] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:26:46,461 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: a2c195c
>2012-12-17 09:26:46,741 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [48bd7fc3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:26:47,035 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [5cb68cfd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:26:47,121 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [5cb68cfd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1b65fc07
>2012-12-17 09:26:47,126 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [5cb68cfd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 24b4d600
>2012-12-17 09:26:47,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [5cb68cfd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:47,278 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [5cb68cfd] FINISH, SpmStopVDSCommand, log id: 24b4d600
>2012-12-17 09:26:47,292 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [5cb68cfd] FINISH, ResetIrsVDSCommand, log id: 1b65fc07
>2012-12-17 09:26:47,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [5cb68cfd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c8b3dea
>2012-12-17 09:26:47,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [5cb68cfd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:47,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [5cb68cfd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c8b3dea
>2012-12-17 09:26:47,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [733324dc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:26:47,384 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [733324dc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4856
>2012-12-17 09:26:47,385 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4856, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2168c598
>2012-12-17 09:26:47,408 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [733324dc] spmStart polling started: taskId = 657e11ff-4690-43cf-bd82-43021a7e4245
>2012-12-17 09:26:48,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [733324dc] spmStart polling ended: taskId = 657e11ff-4690-43cf-bd82-43021a7e4245 task status = finished
>2012-12-17 09:26:48,448 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [733324dc] spmStart polling ended, spm status: SPM
>2012-12-17 09:26:48,450 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=657e11ff-4690-43cf-bd82-43021a7e4245), log id: 772bcdb8
>2012-12-17 09:26:48,468 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, HSMClearTaskVDSCommand, log id: 772bcdb8
>2012-12-17 09:26:48,468 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5ccf2827, log id: 2168c598
>2012-12-17 09:26:48,505 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [733324dc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:26:48,526 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:48,532 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 74fca4a6
>2012-12-17 09:26:48,556 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 74fca4a6
>2012-12-17 09:26:48,556 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a2c195c
>2012-12-17 09:26:48,556 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [733324dc] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:26:48,556 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 776f31ec
>2012-12-17 09:26:48,556 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:48,558 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1510b0a6
>2012-12-17 09:26:48,581 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1510b0a6
>2012-12-17 09:26:48,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 776f31ec
>2012-12-17 09:26:48,581 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [733324dc] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:26:54,894 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [c8c45fe] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:26:57,427 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [7bb72f96] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:26:57,613 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-89) [4f4f0222] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:26:57,691 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5132efe9
>2012-12-17 09:26:57,692 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7d5555af
>2012-12-17 09:26:57,712 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:58,142 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, SpmStopVDSCommand, log id: 7d5555af
>2012-12-17 09:26:58,160 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, ResetIrsVDSCommand, log id: 5132efe9
>2012-12-17 09:26:58,172 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 133428bc
>2012-12-17 09:26:58,172 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4f4f0222] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:26:58,172 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 133428bc
>2012-12-17 09:27:05,101 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:27:08,222 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2a365c8d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:27:08,236 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2a365c8d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4857
>2012-12-17 09:27:08,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4857, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 17f38d43
>2012-12-17 09:27:08,262 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] spmStart polling started: taskId = 4f431e6b-9fba-476e-a5de-b620c6d7bb9e
>2012-12-17 09:27:09,284 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] spmStart polling ended: taskId = 4f431e6b-9fba-476e-a5de-b620c6d7bb9e task status = finished
>2012-12-17 09:27:09,303 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] spmStart polling ended, spm status: SPM
>2012-12-17 09:27:09,304 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4f431e6b-9fba-476e-a5de-b620c6d7bb9e), log id: 58d4409f
>2012-12-17 09:27:09,323 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] FINISH, HSMClearTaskVDSCommand, log id: 58d4409f
>2012-12-17 09:27:09,323 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@56deb448, log id: 17f38d43
>2012-12-17 09:27:09,335 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2a365c8d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:27:09,380 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [3e7d3817] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:27:09,380 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 51e12ba5
>2012-12-17 09:27:09,402 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2a365c8d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:27:09,551 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:27:09,595 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 368222fb
>2012-12-17 09:27:09,596 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 39467503
>2012-12-17 09:27:09,616 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:09,968 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] FINISH, SpmStopVDSCommand, log id: 39467503
>2012-12-17 09:27:10,020 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] FINISH, ResetIrsVDSCommand, log id: 368222fb
>2012-12-17 09:27:10,031 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 10e95887
>2012-12-17 09:27:10,031 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:10,031 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 10e95887
>2012-12-17 09:27:10,081 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:27:10,094 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4858
>2012-12-17 09:27:10,096 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4858, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 39f92493
>2012-12-17 09:27:10,117 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling started: taskId = 52e705fd-8f0c-4443-90a9-8dcd7bb5d720
>2012-12-17 09:27:11,140 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling ended: taskId = 52e705fd-8f0c-4443-90a9-8dcd7bb5d720 task status = finished
>2012-12-17 09:27:11,159 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling ended, spm status: SPM
>2012-12-17 09:27:11,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=52e705fd-8f0c-4443-90a9-8dcd7bb5d720), log id: 23a6ab00
>2012-12-17 09:27:11,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, HSMClearTaskVDSCommand, log id: 23a6ab00
>2012-12-17 09:27:11,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5ff9b88a, log id: 39f92493
>2012-12-17 09:27:11,255 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:27:11,382 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:11,387 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6accbc68
>2012-12-17 09:27:11,407 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6accbc68
>2012-12-17 09:27:11,407 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 51e12ba5
>2012-12-17 09:27:11,407 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [3e7d3817] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:27:11,407 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2765869f
>2012-12-17 09:27:11,407 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:11,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 504ee3c3
>2012-12-17 09:27:11,428 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 504ee3c3
>2012-12-17 09:27:11,428 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2765869f
>2012-12-17 09:27:11,428 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [3e7d3817] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:27:15,365 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:27:20,123 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [4d69cef1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:27:20,264 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [651c3e82] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:27:20,335 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 52d0fb70
>2012-12-17 09:27:20,337 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6023f5d9
>2012-12-17 09:27:20,357 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:20,511 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, SpmStopVDSCommand, log id: 6023f5d9
>2012-12-17 09:27:21,022 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, ResetIrsVDSCommand, log id: 52d0fb70
>2012-12-17 09:27:21,033 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6c6ed2a7
>2012-12-17 09:27:21,034 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [651c3e82] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:21,034 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6c6ed2a7
>2012-12-17 09:27:26,978 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:27:31,237 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7e80f448] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:27:31,444 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7e80f448] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4859
>2012-12-17 09:27:31,445 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4859, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2d6eac23
>2012-12-17 09:27:31,467 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] spmStart polling started: taskId = bc919962-55b5-41a3-8e3a-a8bfd7c4c601
>2012-12-17 09:27:32,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] spmStart polling ended: taskId = bc919962-55b5-41a3-8e3a-a8bfd7c4c601 task status = finished
>2012-12-17 09:27:32,512 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] spmStart polling ended, spm status: SPM
>2012-12-17 09:27:32,513 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bc919962-55b5-41a3-8e3a-a8bfd7c4c601), log id: 60279fc5
>2012-12-17 09:27:32,531 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] FINISH, HSMClearTaskVDSCommand, log id: 60279fc5
>2012-12-17 09:27:32,532 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@496a6fd4, log id: 2d6eac23
>2012-12-17 09:27:32,541 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7e80f448] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:27:32,598 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [7a1c9fc6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:27:32,598 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 42cc891b
>2012-12-17 09:27:32,614 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7e80f448] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:27:32,742 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-34) [65f4d22d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:27:32,786 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 36a0cfa1
>2012-12-17 09:27:32,788 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 242d6a1c
>2012-12-17 09:27:32,808 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:32,928 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] FINISH, SpmStopVDSCommand, log id: 242d6a1c
>2012-12-17 09:27:32,950 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] FINISH, ResetIrsVDSCommand, log id: 36a0cfa1
>2012-12-17 09:27:32,962 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 75461654
>2012-12-17 09:27:32,962 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:32,962 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 75461654
>2012-12-17 09:27:33,035 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:27:33,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4860
>2012-12-17 09:27:33,048 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4860, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 192e9b9e
>2012-12-17 09:27:33,084 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling started: taskId = 1f915299-d10a-4173-96ab-f35b3022cf1c
>2012-12-17 09:27:34,106 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling ended: taskId = 1f915299-d10a-4173-96ab-f35b3022cf1c task status = finished
>2012-12-17 09:27:34,125 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling ended, spm status: SPM
>2012-12-17 09:27:34,126 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1f915299-d10a-4173-96ab-f35b3022cf1c), log id: 7f345cf9
>2012-12-17 09:27:34,144 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, HSMClearTaskVDSCommand, log id: 7f345cf9
>2012-12-17 09:27:34,144 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@46c420ad, log id: 192e9b9e
>2012-12-17 09:27:34,162 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:27:34,180 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:34,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: fc5d975
>2012-12-17 09:27:34,200 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: fc5d975
>2012-12-17 09:27:34,201 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 42cc891b
>2012-12-17 09:27:34,201 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [7a1c9fc6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:27:34,201 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 46675a5e
>2012-12-17 09:27:34,201 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:34,202 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 79510662
>2012-12-17 09:27:34,220 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 79510662
>2012-12-17 09:27:34,221 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 46675a5e
>2012-12-17 09:27:34,221 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [7a1c9fc6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:27:37,209 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:27:43,074 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [94e4f4d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:27:43,209 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [5c496b37] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:27:43,289 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 45ca6579
>2012-12-17 09:27:43,295 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: ca65381
>2012-12-17 09:27:43,325 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:43,494 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, SpmStopVDSCommand, log id: ca65381
>2012-12-17 09:27:43,508 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, ResetIrsVDSCommand, log id: 45ca6579
>2012-12-17 09:27:43,519 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2da3cf6f
>2012-12-17 09:27:43,520 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:43,520 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2da3cf6f
>2012-12-17 09:27:47,476 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [78e614fb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:27:53,598 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [2a7b65] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:27:53,654 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [2a7b65] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4861
>2012-12-17 09:27:53,656 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4861, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 9ec387c
>2012-12-17 09:27:53,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] spmStart polling started: taskId = 19391ac2-88c6-4eaf-802a-deacc72c37db
>2012-12-17 09:27:54,702 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] spmStart polling ended: taskId = 19391ac2-88c6-4eaf-802a-deacc72c37db task status = finished
>2012-12-17 09:27:54,725 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] spmStart polling ended, spm status: SPM
>2012-12-17 09:27:54,727 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=19391ac2-88c6-4eaf-802a-deacc72c37db), log id: 2a6460cb
>2012-12-17 09:27:54,748 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] FINISH, HSMClearTaskVDSCommand, log id: 2a6460cb
>2012-12-17 09:27:54,748 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@240ec52b, log id: 9ec387c
>2012-12-17 09:27:54,805 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [2a7b65] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:27:54,889 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [13d7977] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:27:54,889 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 27b39dc5
>2012-12-17 09:27:54,917 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [2a7b65] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:27:55,106 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-99) [d7157bb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:27:55,165 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [d7157bb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 66f5d546
>2012-12-17 09:27:55,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [d7157bb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5db84d9
>2012-12-17 09:27:55,186 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [d7157bb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:55,331 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [d7157bb] FINISH, SpmStopVDSCommand, log id: 5db84d9
>2012-12-17 09:27:55,346 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [d7157bb] FINISH, ResetIrsVDSCommand, log id: 66f5d546
>2012-12-17 09:27:55,356 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [d7157bb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4889b2bf
>2012-12-17 09:27:55,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [d7157bb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:55,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [d7157bb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4889b2bf
>2012-12-17 09:27:55,396 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [13d7977] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:27:55,423 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [13d7977] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4862
>2012-12-17 09:27:55,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4862, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 68097485
>2012-12-17 09:27:55,446 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [13d7977] spmStart polling started: taskId = 35132a69-ac3b-480a-bfca-2f1b1ff36f4c
>2012-12-17 09:27:56,468 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [13d7977] spmStart polling ended: taskId = 35132a69-ac3b-480a-bfca-2f1b1ff36f4c task status = finished
>2012-12-17 09:27:56,486 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [13d7977] spmStart polling ended, spm status: SPM
>2012-12-17 09:27:56,488 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=35132a69-ac3b-480a-bfca-2f1b1ff36f4c), log id: 440058f5
>2012-12-17 09:27:56,507 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, HSMClearTaskVDSCommand, log id: 440058f5
>2012-12-17 09:27:56,507 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@72d05fb3, log id: 68097485
>2012-12-17 09:27:56,609 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [13d7977] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:27:56,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:56,692 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 53db6f53
>2012-12-17 09:27:56,712 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 53db6f53
>2012-12-17 09:27:56,712 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 27b39dc5
>2012-12-17 09:27:56,712 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [13d7977] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:27:56,712 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6b69dd27
>2012-12-17 09:27:56,712 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:27:56,713 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1b76e567
>2012-12-17 09:27:56,736 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1b76e567
>2012-12-17 09:27:56,736 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6b69dd27
>2012-12-17 09:27:56,736 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [13d7977] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:27:57,727 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [47111e6d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:28:05,437 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [327be49e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:28:06,455 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [6e561ed4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:28:06,521 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 23578fa
>2012-12-17 09:28:06,523 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7123380e
>2012-12-17 09:28:06,544 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:28:06,688 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] FINISH, SpmStopVDSCommand, log id: 7123380e
>2012-12-17 09:28:06,701 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] FINISH, ResetIrsVDSCommand, log id: 23578fa
>2012-12-17 09:28:06,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7aca8471
>2012-12-17 09:28:06,712 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [6e561ed4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:28:06,712 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7aca8471
>2012-12-17 09:28:07,949 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [63fee8f1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:28:16,757 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:28:16,799 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4863
>2012-12-17 09:28:16,800 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4863, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1bf5e53e
>2012-12-17 09:28:16,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling started: taskId = d2640904-9461-4f4b-a96e-0f694acbeb31
>2012-12-17 09:28:17,844 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended: taskId = d2640904-9461-4f4b-a96e-0f694acbeb31 task status = finished
>2012-12-17 09:28:17,863 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended, spm status: SPM
>2012-12-17 09:28:17,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d2640904-9461-4f4b-a96e-0f694acbeb31), log id: 6f7910df
>2012-12-17 09:28:17,882 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMClearTaskVDSCommand, log id: 6f7910df
>2012-12-17 09:28:17,882 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7542d9a5, log id: 1bf5e53e
>2012-12-17 09:28:17,895 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:28:17,941 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:28:17,941 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7f690623
>2012-12-17 09:28:17,978 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:28:18,274 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-54) [2c554bbe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:28:18,332 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 47cdaf9b
>2012-12-17 09:28:18,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 619605e0
>2012-12-17 09:28:18,352 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:28:18,498 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, SpmStopVDSCommand, log id: 619605e0
>2012-12-17 09:28:18,513 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, ResetIrsVDSCommand, log id: 47cdaf9b
>2012-12-17 09:28:18,524 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 10738131
>2012-12-17 09:28:18,524 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:28:18,524 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 10738131
>2012-12-17 09:28:18,575 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:28:18,592 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4864
>2012-12-17 09:28:18,594 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4864, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4286753c
>2012-12-17 09:28:18,616 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling started: taskId = 36317878-b0bf-4593-a56e-c80cfbaa58c5
>2012-12-17 09:28:19,638 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling ended: taskId = 36317878-b0bf-4593-a56e-c80cfbaa58c5 task status = finished
>2012-12-17 09:28:19,656 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling ended, spm status: SPM
>2012-12-17 09:28:19,657 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=36317878-b0bf-4593-a56e-c80cfbaa58c5), log id: 47546308
>2012-12-17 09:28:19,675 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMClearTaskVDSCommand, log id: 47546308
>2012-12-17 09:28:19,675 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7531c6cd, log id: 4286753c
>2012-12-17 09:28:19,693 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:28:19,719 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:28:19,722 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3f05c950
>2012-12-17 09:28:19,749 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3f05c950
>2012-12-17 09:28:19,749 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7f690623
>2012-12-17 09:28:19,749 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:28:19,749 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 52bbe1bd
>2012-12-17 09:28:19,750 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:28:19,751 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 298ed541
>2012-12-17 09:28:19,770 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 298ed541
>2012-12-17 09:28:19,770 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 52bbe1bd
>2012-12-17 09:28:19,770 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:28:28,437 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:28:28,627 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [efbb817] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:28:28,796 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [5c6bc06e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:28:28,851 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [5c6bc06e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: df3cd3b
>2012-12-17 09:28:28,853 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [5c6bc06e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 49aca843
>2012-12-17 09:28:28,872 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [5c6bc06e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:28:29,488 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [5c6bc06e] FINISH, SpmStopVDSCommand, log id: 49aca843
>2012-12-17 09:28:29,506 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [5c6bc06e] FINISH, ResetIrsVDSCommand, log id: df3cd3b
>2012-12-17 09:28:29,517 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [5c6bc06e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7489490c
>2012-12-17 09:28:29,517 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5c6bc06e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:28:29,517 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [5c6bc06e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7489490c
>2012-12-17 09:28:38,682 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:28:39,629 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5d07077e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:28:39,640 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5d07077e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4865
>2012-12-17 09:28:39,641 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4865, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 31ac78f
>2012-12-17 09:28:39,663 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] spmStart polling started: taskId = cb8c0197-0618-4096-b0fd-5f2555f6c005
>2012-12-17 09:28:40,685 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] spmStart polling ended: taskId = cb8c0197-0618-4096-b0fd-5f2555f6c005 task status = finished
>2012-12-17 09:28:40,706 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] spmStart polling ended, spm status: SPM
>2012-12-17 09:28:40,707 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cb8c0197-0618-4096-b0fd-5f2555f6c005), log id: 213b0f3d
>2012-12-17 09:28:40,737 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] FINISH, HSMClearTaskVDSCommand, log id: 213b0f3d
>2012-12-17 09:28:40,737 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2c657ed6, log id: 31ac78f
>2012-12-17 09:28:41,189 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5d07077e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:28:41,251 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-38) [5389729] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:28:41,252 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5fb11eb6
>2012-12-17 09:28:41,277 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5d07077e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:28:41,402 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [be98275] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:28:41,450 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [be98275] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4cabb600
>2012-12-17 09:28:41,451 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [be98275] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 73ebdae8
>2012-12-17 09:28:41,471 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [be98275] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:28:42,213 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [be98275] FINISH, SpmStopVDSCommand, log id: 73ebdae8
>2012-12-17 09:28:42,235 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [be98275] FINISH, ResetIrsVDSCommand, log id: 4cabb600
>2012-12-17 09:28:42,246 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [be98275] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 705c5f97
>2012-12-17 09:28:42,246 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [be98275] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:28:42,246 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [be98275] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 705c5f97
>2012-12-17 09:28:42,289 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [5389729] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:28:42,302 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [5389729] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4866
>2012-12-17 09:28:42,303 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4866, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4b49b323
>2012-12-17 09:28:42,332 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [5389729] spmStart polling started: taskId = faa598b3-15c3-48ea-9546-d744dc26eeb7
>2012-12-17 09:28:43,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [5389729] spmStart polling ended: taskId = faa598b3-15c3-48ea-9546-d744dc26eeb7 task status = finished
>2012-12-17 09:28:43,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [5389729] spmStart polling ended, spm status: SPM
>2012-12-17 09:28:43,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=faa598b3-15c3-48ea-9546-d744dc26eeb7), log id: def2b43
>2012-12-17 09:28:43,393 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, HSMClearTaskVDSCommand, log id: def2b43
>2012-12-17 09:28:43,393 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@510142da, log id: 4b49b323
>2012-12-17 09:28:43,420 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [5389729] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:28:43,478 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:28:43,483 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 23554bb6
>2012-12-17 09:28:43,502 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 23554bb6
>2012-12-17 09:28:43,502 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5fb11eb6
>2012-12-17 09:28:43,502 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-38) [5389729] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:28:43,502 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 34ddabd9
>2012-12-17 09:28:43,502 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:28:43,503 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 602e5ddf
>2012-12-17 09:28:43,522 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 602e5ddf
>2012-12-17 09:28:43,522 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 34ddabd9
>2012-12-17 09:28:43,522 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-38) [5389729] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:28:48,930 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [95f0e17] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:28:52,331 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:28:52,431 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-95) [136706cf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:28:52,502 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-95) [136706cf] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6af4f54b
>2012-12-17 09:28:52,503 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [136706cf] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 336ed08c
>2012-12-17 09:28:52,523 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [136706cf] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:28:52,691 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [136706cf] FINISH, SpmStopVDSCommand, log id: 336ed08c
>2012-12-17 09:28:52,705 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-95) [136706cf] FINISH, ResetIrsVDSCommand, log id: 6af4f54b
>2012-12-17 09:28:52,716 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [136706cf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 482dc65c
>2012-12-17 09:28:52,716 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [136706cf] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:28:52,716 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [136706cf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 482dc65c
>2012-12-17 09:28:59,109 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [622cf65b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:29:02,771 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:29:02,791 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4867
>2012-12-17 09:29:02,792 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4867, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 59da520c
>2012-12-17 09:29:02,815 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] spmStart polling started: taskId = f9ee22a5-4d03-41cf-be07-f912a5ee33c4
>2012-12-17 09:29:03,836 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] spmStart polling ended: taskId = f9ee22a5-4d03-41cf-be07-f912a5ee33c4 task status = finished
>2012-12-17 09:29:03,854 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] spmStart polling ended, spm status: SPM
>2012-12-17 09:29:03,856 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f9ee22a5-4d03-41cf-be07-f912a5ee33c4), log id: a741db6
>2012-12-17 09:29:03,874 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, HSMClearTaskVDSCommand, log id: a741db6
>2012-12-17 09:29:03,874 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4d4a9d16, log id: 59da520c
>2012-12-17 09:29:03,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:29:03,931 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [160c510e] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:29:03,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 9a2a5e7
>2012-12-17 09:29:03,948 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:29:04,076 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [56a0d824] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:29:04,144 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 269beee5
>2012-12-17 09:29:04,145 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2113d277
>2012-12-17 09:29:04,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:04,312 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] FINISH, SpmStopVDSCommand, log id: 2113d277
>2012-12-17 09:29:04,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] FINISH, ResetIrsVDSCommand, log id: 269beee5
>2012-12-17 09:29:04,339 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 394264f4
>2012-12-17 09:29:04,339 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [56a0d824] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:04,339 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 394264f4
>2012-12-17 09:29:04,388 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:29:04,397 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4868
>2012-12-17 09:29:04,398 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4868, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6d25b0fb
>2012-12-17 09:29:04,421 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] spmStart polling started: taskId = 1f705429-cb32-476b-a594-048a9601f738
>2012-12-17 09:29:05,442 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] spmStart polling ended: taskId = 1f705429-cb32-476b-a594-048a9601f738 task status = finished
>2012-12-17 09:29:05,460 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] spmStart polling ended, spm status: SPM
>2012-12-17 09:29:05,462 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1f705429-cb32-476b-a594-048a9601f738), log id: 33776c1a
>2012-12-17 09:29:05,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, HSMClearTaskVDSCommand, log id: 33776c1a
>2012-12-17 09:29:05,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@655a395b, log id: 6d25b0fb
>2012-12-17 09:29:06,093 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:29:06,288 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:06,290 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6896349a
>2012-12-17 09:29:06,308 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6896349a
>2012-12-17 09:29:06,308 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 9a2a5e7
>2012-12-17 09:29:06,308 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [160c510e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:29:06,309 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5d57a01a
>2012-12-17 09:29:06,309 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:06,310 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 37d762fc
>2012-12-17 09:29:06,328 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 37d762fc
>2012-12-17 09:29:06,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5d57a01a
>2012-12-17 09:29:06,328 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [160c510e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:29:09,285 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:29:14,468 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:29:14,651 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [7a6ed577] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:29:14,731 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 72c5c564
>2012-12-17 09:29:14,732 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3e12c6bf
>2012-12-17 09:29:14,752 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:14,938 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, SpmStopVDSCommand, log id: 3e12c6bf
>2012-12-17 09:29:14,950 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, ResetIrsVDSCommand, log id: 72c5c564
>2012-12-17 09:29:14,961 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 74b98e56
>2012-12-17 09:29:14,961 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:14,961 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 74b98e56
>2012-12-17 09:29:19,549 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:29:25,018 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:29:25,033 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4869
>2012-12-17 09:29:25,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4869, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 359e225d
>2012-12-17 09:29:25,058 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] spmStart polling started: taskId = 876570e6-055f-4b7e-8e1d-b6ebfcc2d988
>2012-12-17 09:29:26,081 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] spmStart polling ended: taskId = 876570e6-055f-4b7e-8e1d-b6ebfcc2d988 task status = finished
>2012-12-17 09:29:26,100 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] spmStart polling ended, spm status: SPM
>2012-12-17 09:29:26,101 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=876570e6-055f-4b7e-8e1d-b6ebfcc2d988), log id: 4248830a
>2012-12-17 09:29:26,120 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, HSMClearTaskVDSCommand, log id: 4248830a
>2012-12-17 09:29:26,120 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@26932785, log id: 359e225d
>2012-12-17 09:29:26,537 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:29:26,711 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [7790038] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:29:26,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 311ed19c
>2012-12-17 09:29:26,761 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:29:26,857 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [280f54d2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:29:26,926 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17716e9c
>2012-12-17 09:29:26,928 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 15995375
>2012-12-17 09:29:26,947 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:27,474 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] FINISH, SpmStopVDSCommand, log id: 15995375
>2012-12-17 09:29:27,493 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] FINISH, ResetIrsVDSCommand, log id: 17716e9c
>2012-12-17 09:29:27,504 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2fff36b2
>2012-12-17 09:29:27,504 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [280f54d2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:27,504 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2fff36b2
>2012-12-17 09:29:27,551 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:29:27,580 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4870
>2012-12-17 09:29:27,582 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4870, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3f70bbe2
>2012-12-17 09:29:27,603 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] spmStart polling started: taskId = ce30fad2-2c85-42bf-a79a-6fe585a2b296
>2012-12-17 09:29:28,626 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] spmStart polling ended: taskId = ce30fad2-2c85-42bf-a79a-6fe585a2b296 task status = finished
>2012-12-17 09:29:28,645 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] spmStart polling ended, spm status: SPM
>2012-12-17 09:29:28,646 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ce30fad2-2c85-42bf-a79a-6fe585a2b296), log id: 72ca5e04
>2012-12-17 09:29:28,665 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, HSMClearTaskVDSCommand, log id: 72ca5e04
>2012-12-17 09:29:28,665 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@43e2937c, log id: 3f70bbe2
>2012-12-17 09:29:28,677 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:29:28,718 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:28,724 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 383ec615
>2012-12-17 09:29:28,743 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 383ec615
>2012-12-17 09:29:28,743 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 311ed19c
>2012-12-17 09:29:28,744 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [7790038] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:29:28,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 213ea98c
>2012-12-17 09:29:28,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:28,745 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25e52a39
>2012-12-17 09:29:28,764 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25e52a39
>2012-12-17 09:29:28,764 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 213ea98c
>2012-12-17 09:29:28,764 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [7790038] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:29:29,735 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:29:37,587 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3aebafd8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:29:37,734 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [42560f9e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:29:37,820 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 65232a6a
>2012-12-17 09:29:37,821 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2238829b
>2012-12-17 09:29:37,844 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:38,010 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, SpmStopVDSCommand, log id: 2238829b
>2012-12-17 09:29:38,088 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, ResetIrsVDSCommand, log id: 65232a6a
>2012-12-17 09:29:38,098 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6d17e67b
>2012-12-17 09:29:38,099 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:38,099 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6d17e67b
>2012-12-17 09:29:39,973 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1ed39251] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:29:48,172 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [1cfebcc3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:29:48,189 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [1cfebcc3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4871
>2012-12-17 09:29:48,190 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4871, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2dd8977e
>2012-12-17 09:29:48,213 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] spmStart polling started: taskId = f1d34bf6-dadb-4387-ae8f-f72a91918a57
>2012-12-17 09:29:49,235 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] spmStart polling ended: taskId = f1d34bf6-dadb-4387-ae8f-f72a91918a57 task status = finished
>2012-12-17 09:29:49,253 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] spmStart polling ended, spm status: SPM
>2012-12-17 09:29:49,254 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f1d34bf6-dadb-4387-ae8f-f72a91918a57), log id: 29f52a31
>2012-12-17 09:29:49,273 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] FINISH, HSMClearTaskVDSCommand, log id: 29f52a31
>2012-12-17 09:29:49,273 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1ea50c7c, log id: 2dd8977e
>2012-12-17 09:29:49,291 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [1cfebcc3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:29:49,343 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:29:49,344 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 37d1bbbf
>2012-12-17 09:29:49,357 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [1cfebcc3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:29:49,500 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-51) [6c33f0df] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:29:49,547 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 35eed4b4
>2012-12-17 09:29:49,549 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3bbe9688
>2012-12-17 09:29:49,569 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:49,728 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] FINISH, SpmStopVDSCommand, log id: 3bbe9688
>2012-12-17 09:29:49,738 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] FINISH, ResetIrsVDSCommand, log id: 35eed4b4
>2012-12-17 09:29:49,749 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 16768ea2
>2012-12-17 09:29:49,749 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [6c33f0df] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:49,749 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 16768ea2
>2012-12-17 09:29:49,822 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:29:49,834 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4872
>2012-12-17 09:29:49,837 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4872, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 15e2efb3
>2012-12-17 09:29:49,861 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling started: taskId = a3230d65-581c-4547-a8b7-e43ff39d0532
>2012-12-17 09:29:50,882 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended: taskId = a3230d65-581c-4547-a8b7-e43ff39d0532 task status = finished
>2012-12-17 09:29:50,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended, spm status: SPM
>2012-12-17 09:29:50,901 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a3230d65-581c-4547-a8b7-e43ff39d0532), log id: 180316fc
>2012-12-17 09:29:50,920 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMClearTaskVDSCommand, log id: 180316fc
>2012-12-17 09:29:50,920 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@72da06b4, log id: 15e2efb3
>2012-12-17 09:29:51,433 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:29:51,557 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:51,559 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 459aff85
>2012-12-17 09:29:51,578 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 459aff85
>2012-12-17 09:29:51,578 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 37d1bbbf
>2012-12-17 09:29:51,578 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:29:51,578 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 42fd9876
>2012-12-17 09:29:51,578 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:29:51,579 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5362a7ed
>2012-12-17 09:29:51,598 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5362a7ed
>2012-12-17 09:29:51,598 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 42fd9876
>2012-12-17 09:29:51,598 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:29:59,861 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [733324dc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:30:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [3a1c2706] Checking autorecoverable hosts
>2012-12-17 09:30:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [3a1c2706] Autorecovering 0 hosts
>2012-12-17 09:30:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [3a1c2706] Checking autorecoverable hosts done
>2012-12-17 09:30:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [3a1c2706] Autorecovering storage domains is disabled, skipping
>2012-12-17 09:30:00,030 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-56) [6de308ea] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:30:00,127 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7d973d6f
>2012-12-17 09:30:00,129 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 189253e9
>2012-12-17 09:30:00,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:00,301 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] FINISH, SpmStopVDSCommand, log id: 189253e9
>2012-12-17 09:30:00,321 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] FINISH, ResetIrsVDSCommand, log id: 7d973d6f
>2012-12-17 09:30:00,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2f20907d
>2012-12-17 09:30:00,392 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6de308ea] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:00,392 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2f20907d
>2012-12-17 09:30:00,611 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:30:11,019 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [223c8c2d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:30:11,489 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [223c8c2d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4873
>2012-12-17 09:30:11,491 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4873, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2deba123
>2012-12-17 09:30:11,515 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] spmStart polling started: taskId = ef3b1a6c-e028-4bf9-b681-e4f8fe1a9b7e
>2012-12-17 09:30:12,536 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] spmStart polling ended: taskId = ef3b1a6c-e028-4bf9-b681-e4f8fe1a9b7e task status = finished
>2012-12-17 09:30:12,555 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] spmStart polling ended, spm status: SPM
>2012-12-17 09:30:12,556 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ef3b1a6c-e028-4bf9-b681-e4f8fe1a9b7e), log id: 7226118a
>2012-12-17 09:30:12,574 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] FINISH, HSMClearTaskVDSCommand, log id: 7226118a
>2012-12-17 09:30:12,575 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2149b592, log id: 2deba123
>2012-12-17 09:30:12,588 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [223c8c2d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:30:12,627 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [20c0de80] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:30:12,627 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1ebf4809
>2012-12-17 09:30:12,665 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [223c8c2d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:30:12,755 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [1813139a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:30:12,807 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [1813139a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 71ba17fd
>2012-12-17 09:30:12,810 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [1813139a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26a4fc03
>2012-12-17 09:30:12,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [1813139a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:13,193 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [1813139a] FINISH, SpmStopVDSCommand, log id: 26a4fc03
>2012-12-17 09:30:13,205 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [1813139a] FINISH, ResetIrsVDSCommand, log id: 71ba17fd
>2012-12-17 09:30:13,216 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [1813139a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4d9d0960
>2012-12-17 09:30:13,216 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [1813139a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:13,216 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [1813139a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4d9d0960
>2012-12-17 09:30:13,268 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [20c0de80] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:30:13,288 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [20c0de80] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4874
>2012-12-17 09:30:13,289 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4874, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: a927837
>2012-12-17 09:30:13,312 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] spmStart polling started: taskId = 4d2aa85d-ba88-4a87-b26d-4c01694dd3c7
>2012-12-17 09:30:14,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] spmStart polling ended: taskId = 4d2aa85d-ba88-4a87-b26d-4c01694dd3c7 task status = finished
>2012-12-17 09:30:14,351 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] spmStart polling ended, spm status: SPM
>2012-12-17 09:30:14,352 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4d2aa85d-ba88-4a87-b26d-4c01694dd3c7), log id: 48053e9c
>2012-12-17 09:30:14,370 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, HSMClearTaskVDSCommand, log id: 48053e9c
>2012-12-17 09:30:14,370 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@505198b4, log id: a927837
>2012-12-17 09:30:14,382 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [20c0de80] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:30:14,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:14,403 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 10332a07
>2012-12-17 09:30:14,422 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 10332a07
>2012-12-17 09:30:14,422 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1ebf4809
>2012-12-17 09:30:14,422 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [20c0de80] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:30:14,422 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6baba8d7
>2012-12-17 09:30:14,422 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:14,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5c804efb
>2012-12-17 09:30:14,442 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5c804efb
>2012-12-17 09:30:14,442 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6baba8d7
>2012-12-17 09:30:14,442 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [20c0de80] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:30:22,153 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [47111e6d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:30:23,313 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [10684b21] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:30:23,455 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-45) [7db92791] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:30:23,524 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7614816d
>2012-12-17 09:30:23,526 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 44a4be41
>2012-12-17 09:30:23,546 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [7db92791] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:23,673 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, SpmStopVDSCommand, log id: 44a4be41
>2012-12-17 09:30:23,692 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, ResetIrsVDSCommand, log id: 7614816d
>2012-12-17 09:30:23,703 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 31bb0dbb
>2012-12-17 09:30:23,703 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7db92791] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:23,703 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 31bb0dbb
>2012-12-17 09:30:32,363 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:30:33,805 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3c6804c8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:30:33,819 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3c6804c8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4875
>2012-12-17 09:30:33,820 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4875, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 101d8222
>2012-12-17 09:30:33,843 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] spmStart polling started: taskId = dacd1950-5ed6-4ac7-a7ef-8d6ca9b72534
>2012-12-17 09:30:34,865 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] spmStart polling ended: taskId = dacd1950-5ed6-4ac7-a7ef-8d6ca9b72534 task status = finished
>2012-12-17 09:30:34,884 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] spmStart polling ended, spm status: SPM
>2012-12-17 09:30:34,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dacd1950-5ed6-4ac7-a7ef-8d6ca9b72534), log id: ea3dbe0
>2012-12-17 09:30:34,903 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] FINISH, HSMClearTaskVDSCommand, log id: ea3dbe0
>2012-12-17 09:30:34,903 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@27b856a2, log id: 101d8222
>2012-12-17 09:30:34,930 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3c6804c8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:30:35,015 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-41) [26c9f399] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:30:35,016 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5588918a
>2012-12-17 09:30:35,030 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3c6804c8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:30:35,154 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [603633ae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:30:35,238 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [603633ae] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1008b992
>2012-12-17 09:30:35,240 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [603633ae] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 49c61278
>2012-12-17 09:30:35,260 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [603633ae] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:35,416 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [603633ae] FINISH, SpmStopVDSCommand, log id: 49c61278
>2012-12-17 09:30:35,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [603633ae] FINISH, ResetIrsVDSCommand, log id: 1008b992
>2012-12-17 09:30:35,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [603633ae] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: d29ac82
>2012-12-17 09:30:35,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [603633ae] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:35,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [603633ae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: d29ac82
>2012-12-17 09:30:36,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [26c9f399] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:30:36,526 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [26c9f399] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4876
>2012-12-17 09:30:36,528 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4876, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e42a310
>2012-12-17 09:30:36,550 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] spmStart polling started: taskId = 70362b6b-0b4d-438f-a1d1-2ddcbe489210
>2012-12-17 09:30:37,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] spmStart polling ended: taskId = 70362b6b-0b4d-438f-a1d1-2ddcbe489210 task status = finished
>2012-12-17 09:30:37,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] spmStart polling ended, spm status: SPM
>2012-12-17 09:30:37,597 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=70362b6b-0b4d-438f-a1d1-2ddcbe489210), log id: 57daa2ec
>2012-12-17 09:30:37,615 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, HSMClearTaskVDSCommand, log id: 57daa2ec
>2012-12-17 09:30:37,615 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@52fe6103, log id: 5e42a310
>2012-12-17 09:30:37,623 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [26c9f399] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:30:37,651 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:37,653 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4de540eb
>2012-12-17 09:30:37,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4de540eb
>2012-12-17 09:30:37,675 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5588918a
>2012-12-17 09:30:37,675 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-41) [26c9f399] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:30:37,675 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6092c41
>2012-12-17 09:30:37,675 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:37,677 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2db24135
>2012-12-17 09:30:37,696 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2db24135
>2012-12-17 09:30:37,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6092c41
>2012-12-17 09:30:37,696 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-41) [26c9f399] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:30:42,650 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:30:46,445 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [5cb68cfd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:30:46,811 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [462eb8a5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:30:46,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 817854
>2012-12-17 09:30:46,863 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 27b1e2e1
>2012-12-17 09:30:46,882 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:47,127 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, SpmStopVDSCommand, log id: 27b1e2e1
>2012-12-17 09:30:47,141 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, ResetIrsVDSCommand, log id: 817854
>2012-12-17 09:30:47,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 575602c0
>2012-12-17 09:30:47,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:47,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 575602c0
>2012-12-17 09:30:52,886 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [29b31be1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:30:57,198 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [550a1bf7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:30:57,211 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [550a1bf7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4877
>2012-12-17 09:30:57,213 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4877, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 50984a60
>2012-12-17 09:30:57,235 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] spmStart polling started: taskId = 0cf67029-1890-40be-aa40-c97caeae5775
>2012-12-17 09:30:58,257 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] spmStart polling ended: taskId = 0cf67029-1890-40be-aa40-c97caeae5775 task status = finished
>2012-12-17 09:30:58,274 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] spmStart polling ended, spm status: SPM
>2012-12-17 09:30:58,275 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0cf67029-1890-40be-aa40-c97caeae5775), log id: 2b980dea
>2012-12-17 09:30:58,293 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] FINISH, HSMClearTaskVDSCommand, log id: 2b980dea
>2012-12-17 09:30:58,293 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@bb99402, log id: 50984a60
>2012-12-17 09:30:58,305 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [550a1bf7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:30:58,358 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-22) [6e84ec96] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:30:58,358 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4ae4cf34
>2012-12-17 09:30:58,407 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [550a1bf7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:30:58,600 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [e8edb34] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:30:58,661 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [e8edb34] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 76bdbdd2
>2012-12-17 09:30:58,662 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [e8edb34] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5cccfd6f
>2012-12-17 09:30:58,682 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [e8edb34] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:58,842 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [e8edb34] FINISH, SpmStopVDSCommand, log id: 5cccfd6f
>2012-12-17 09:30:58,857 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [e8edb34] FINISH, ResetIrsVDSCommand, log id: 76bdbdd2
>2012-12-17 09:30:58,868 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [e8edb34] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49db2046
>2012-12-17 09:30:58,868 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [e8edb34] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:30:58,868 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [e8edb34] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49db2046
>2012-12-17 09:30:58,914 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:30:58,939 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4878
>2012-12-17 09:30:58,940 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4878, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 140841e7
>2012-12-17 09:30:58,966 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] spmStart polling started: taskId = d9592c81-084f-4fd1-9b33-6f9a2c182860
>2012-12-17 09:30:59,988 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] spmStart polling ended: taskId = d9592c81-084f-4fd1-9b33-6f9a2c182860 task status = finished
>2012-12-17 09:31:00,006 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] spmStart polling ended, spm status: SPM
>2012-12-17 09:31:00,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d9592c81-084f-4fd1-9b33-6f9a2c182860), log id: 6997a10a
>2012-12-17 09:31:00,026 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, HSMClearTaskVDSCommand, log id: 6997a10a
>2012-12-17 09:31:00,026 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@61cdcdae, log id: 140841e7
>2012-12-17 09:31:00,052 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:31:00,075 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:00,077 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 68d89b75
>2012-12-17 09:31:00,095 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 68d89b75
>2012-12-17 09:31:00,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4ae4cf34
>2012-12-17 09:31:00,095 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-22) [6e84ec96] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:31:00,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 64c65419
>2012-12-17 09:31:00,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:00,096 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6935c578
>2012-12-17 09:31:00,114 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6935c578
>2012-12-17 09:31:00,114 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 64c65419
>2012-12-17 09:31:00,114 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-22) [6e84ec96] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:31:03,083 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:31:08,964 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:31:09,091 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [6157bb5e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:31:09,151 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6a62ce1e
>2012-12-17 09:31:09,153 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 74b5e2fe
>2012-12-17 09:31:09,173 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:09,913 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, SpmStopVDSCommand, log id: 74b5e2fe
>2012-12-17 09:31:09,940 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, ResetIrsVDSCommand, log id: 6a62ce1e
>2012-12-17 09:31:09,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1ee643e2
>2012-12-17 09:31:09,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [6157bb5e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:09,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1ee643e2
>2012-12-17 09:31:13,262 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:31:20,040 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:31:20,083 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4879
>2012-12-17 09:31:20,084 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4879, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4ea7a045
>2012-12-17 09:31:20,106 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] spmStart polling started: taskId = 18c9d7a0-76bc-412e-b3e0-e227ba7fae9d
>2012-12-17 09:31:21,128 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] spmStart polling ended: taskId = 18c9d7a0-76bc-412e-b3e0-e227ba7fae9d task status = finished
>2012-12-17 09:31:21,146 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] spmStart polling ended, spm status: SPM
>2012-12-17 09:31:21,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=18c9d7a0-76bc-412e-b3e0-e227ba7fae9d), log id: 6ac424e1
>2012-12-17 09:31:21,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, HSMClearTaskVDSCommand, log id: 6ac424e1
>2012-12-17 09:31:21,166 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@73516689, log id: 4ea7a045
>2012-12-17 09:31:21,268 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:31:21,362 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [e893ae] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:31:21,362 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2da985ca
>2012-12-17 09:31:21,543 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:31:21,694 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [48ba699e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:31:21,756 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [48ba699e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 14c2c8b6
>2012-12-17 09:31:21,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [48ba699e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 37e782b5
>2012-12-17 09:31:21,777 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [48ba699e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:21,907 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [48ba699e] FINISH, SpmStopVDSCommand, log id: 37e782b5
>2012-12-17 09:31:21,923 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [48ba699e] FINISH, ResetIrsVDSCommand, log id: 14c2c8b6
>2012-12-17 09:31:21,935 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [48ba699e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ecdd5bf
>2012-12-17 09:31:21,935 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [48ba699e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:21,935 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [48ba699e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ecdd5bf
>2012-12-17 09:31:21,986 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:31:22,000 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4880
>2012-12-17 09:31:22,002 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4880, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7125c62a
>2012-12-17 09:31:22,025 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] spmStart polling started: taskId = 8458759d-4f5f-4d78-8a35-ed0b81652baf
>2012-12-17 09:31:23,048 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] spmStart polling ended: taskId = 8458759d-4f5f-4d78-8a35-ed0b81652baf task status = finished
>2012-12-17 09:31:23,069 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] spmStart polling ended, spm status: SPM
>2012-12-17 09:31:23,070 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8458759d-4f5f-4d78-8a35-ed0b81652baf), log id: 7fa93d7b
>2012-12-17 09:31:23,089 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, HSMClearTaskVDSCommand, log id: 7fa93d7b
>2012-12-17 09:31:23,089 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@400397b7, log id: 7125c62a
>2012-12-17 09:31:23,097 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:31:23,119 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:23,120 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 947346a
>2012-12-17 09:31:23,141 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 947346a
>2012-12-17 09:31:23,141 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2da985ca
>2012-12-17 09:31:23,141 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [e893ae] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:31:23,141 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4fbcc9a8
>2012-12-17 09:31:23,141 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:23,142 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3a9c2c80
>2012-12-17 09:31:23,161 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3a9c2c80
>2012-12-17 09:31:23,161 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4fbcc9a8
>2012-12-17 09:31:23,161 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [e893ae] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:31:23,474 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:31:32,038 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [45dceb6a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:31:32,232 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [134ba83e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:31:32,296 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 445553cc
>2012-12-17 09:31:32,297 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 61c38280
>2012-12-17 09:31:32,317 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:32,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, SpmStopVDSCommand, log id: 61c38280
>2012-12-17 09:31:32,520 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, ResetIrsVDSCommand, log id: 445553cc
>2012-12-17 09:31:32,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1fe58d95
>2012-12-17 09:31:32,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [134ba83e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:32,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1fe58d95
>2012-12-17 09:31:33,672 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6b59f114] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:31:42,578 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [c8c45fe] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:31:42,637 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [c8c45fe] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4881
>2012-12-17 09:31:42,638 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4881, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2d0484f9
>2012-12-17 09:31:42,660 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] spmStart polling started: taskId = 06cbe60a-038d-4526-8e13-73490e871d33
>2012-12-17 09:31:43,683 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] spmStart polling ended: taskId = 06cbe60a-038d-4526-8e13-73490e871d33 task status = finished
>2012-12-17 09:31:43,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] spmStart polling ended, spm status: SPM
>2012-12-17 09:31:43,703 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=06cbe60a-038d-4526-8e13-73490e871d33), log id: 6f98481c
>2012-12-17 09:31:43,721 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] FINISH, HSMClearTaskVDSCommand, log id: 6f98481c
>2012-12-17 09:31:43,722 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@76a65f80, log id: 2d0484f9
>2012-12-17 09:31:43,736 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [c8c45fe] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:31:43,779 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1240becc] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:31:43,779 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4bbbb5e5
>2012-12-17 09:31:43,798 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [c8c45fe] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:31:43,986 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-75) [3da48235] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:31:44,036 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [3da48235] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 24a263f1
>2012-12-17 09:31:44,037 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [3da48235] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 23a4ca5f
>2012-12-17 09:31:44,057 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [3da48235] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:44,640 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [3da48235] FINISH, SpmStopVDSCommand, log id: 23a4ca5f
>2012-12-17 09:31:44,660 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [3da48235] FINISH, ResetIrsVDSCommand, log id: 24a263f1
>2012-12-17 09:31:44,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [3da48235] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40994e0a
>2012-12-17 09:31:44,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [3da48235] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:44,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [3da48235] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40994e0a
>2012-12-17 09:31:44,743 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:31:44,763 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4882
>2012-12-17 09:31:44,764 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4882, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4a213bc7
>2012-12-17 09:31:44,787 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling started: taskId = eb9954f3-ab63-4574-9918-4507fd7bec02
>2012-12-17 09:31:45,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling ended: taskId = eb9954f3-ab63-4574-9918-4507fd7bec02 task status = finished
>2012-12-17 09:31:45,827 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling ended, spm status: SPM
>2012-12-17 09:31:45,829 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=eb9954f3-ab63-4574-9918-4507fd7bec02), log id: 233f43ce
>2012-12-17 09:31:45,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, HSMClearTaskVDSCommand, log id: 233f43ce
>2012-12-17 09:31:45,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e7c9b76, log id: 4a213bc7
>2012-12-17 09:31:45,954 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:31:46,075 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:46,079 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4a6445f0
>2012-12-17 09:31:46,100 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4a6445f0
>2012-12-17 09:31:46,100 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4bbbb5e5
>2012-12-17 09:31:46,100 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1240becc] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:31:46,100 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7d2bb5b7
>2012-12-17 09:31:46,100 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:46,101 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 649610b6
>2012-12-17 09:31:46,119 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 649610b6
>2012-12-17 09:31:46,119 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7d2bb5b7
>2012-12-17 09:31:46,120 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1240becc] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:31:54,156 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:31:54,782 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [5389729] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:31:54,942 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [23d69744] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:31:55,097 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [23d69744] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4162f15e
>2012-12-17 09:31:55,099 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [23d69744] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 555bc2a2
>2012-12-17 09:31:55,125 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [23d69744] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:55,796 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [23d69744] FINISH, SpmStopVDSCommand, log id: 555bc2a2
>2012-12-17 09:31:56,464 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [23d69744] FINISH, ResetIrsVDSCommand, log id: 4162f15e
>2012-12-17 09:31:56,475 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [23d69744] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6be3c5b3
>2012-12-17 09:31:56,475 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [23d69744] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:31:56,475 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [23d69744] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6be3c5b3
>2012-12-17 09:32:04,333 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:32:06,788 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:32:06,812 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4883
>2012-12-17 09:32:06,813 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4883, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4576deca
>2012-12-17 09:32:06,836 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] spmStart polling started: taskId = d7bd7d11-2850-44e4-b78f-0027d0df2adf
>2012-12-17 09:32:07,858 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] spmStart polling ended: taskId = d7bd7d11-2850-44e4-b78f-0027d0df2adf task status = finished
>2012-12-17 09:32:07,877 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] spmStart polling ended, spm status: SPM
>2012-12-17 09:32:07,878 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d7bd7d11-2850-44e4-b78f-0027d0df2adf), log id: 6b51eaef
>2012-12-17 09:32:07,897 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, HSMClearTaskVDSCommand, log id: 6b51eaef
>2012-12-17 09:32:07,897 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f1727f4, log id: 4576deca
>2012-12-17 09:32:07,911 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:32:07,960 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [3fe36f5] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:32:07,960 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5b7c75dd
>2012-12-17 09:32:08,007 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:32:08,196 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [5d1b6c83] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:32:08,274 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 12dde100
>2012-12-17 09:32:08,276 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 791353dd
>2012-12-17 09:32:08,297 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:08,899 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] FINISH, SpmStopVDSCommand, log id: 791353dd
>2012-12-17 09:32:08,916 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] FINISH, ResetIrsVDSCommand, log id: 12dde100
>2012-12-17 09:32:08,927 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 68c48d73
>2012-12-17 09:32:08,927 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5d1b6c83] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:08,927 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 68c48d73
>2012-12-17 09:32:08,996 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3fe36f5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:32:09,020 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3fe36f5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4884
>2012-12-17 09:32:09,021 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4884, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3a5ec553
>2012-12-17 09:32:09,044 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] spmStart polling started: taskId = cfad4488-5339-43bc-bfef-efa7bcbe59ca
>2012-12-17 09:32:10,066 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] spmStart polling ended: taskId = cfad4488-5339-43bc-bfef-efa7bcbe59ca task status = finished
>2012-12-17 09:32:10,086 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] spmStart polling ended, spm status: SPM
>2012-12-17 09:32:10,088 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cfad4488-5339-43bc-bfef-efa7bcbe59ca), log id: 373ed2e1
>2012-12-17 09:32:10,106 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, HSMClearTaskVDSCommand, log id: 373ed2e1
>2012-12-17 09:32:10,106 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7d86ce1c, log id: 3a5ec553
>2012-12-17 09:32:10,122 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3fe36f5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:32:10,147 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:10,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 441b6e47
>2012-12-17 09:32:10,168 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 441b6e47
>2012-12-17 09:32:10,168 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5b7c75dd
>2012-12-17 09:32:10,168 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [3fe36f5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:32:10,169 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 64e51a3f
>2012-12-17 09:32:10,169 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:10,170 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2e7ac27
>2012-12-17 09:32:10,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2e7ac27
>2012-12-17 09:32:10,189 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 64e51a3f
>2012-12-17 09:32:10,189 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [3fe36f5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:32:14,546 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [40c8f45e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:32:19,037 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [3b88f2b5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:32:19,140 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [302432ce] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:32:19,178 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4a9b892e
>2012-12-17 09:32:19,180 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: bbe5cc2
>2012-12-17 09:32:19,200 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [302432ce] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:19,593 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SpmStopVDSCommand, log id: bbe5cc2
>2012-12-17 09:32:19,611 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, ResetIrsVDSCommand, log id: 4a9b892e
>2012-12-17 09:32:19,629 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: d3e7bef
>2012-12-17 09:32:19,629 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:19,630 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, MarkPoolInReconstructModeVDSCommand, log id: d3e7bef
>2012-12-17 09:32:24,761 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6de308ea] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:32:29,677 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:32:29,703 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4885
>2012-12-17 09:32:29,705 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4885, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 58ee0f0c
>2012-12-17 09:32:29,726 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] spmStart polling started: taskId = ab5ab054-c822-49ec-9ae0-62ec47da7210
>2012-12-17 09:32:30,748 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] spmStart polling ended: taskId = ab5ab054-c822-49ec-9ae0-62ec47da7210 task status = finished
>2012-12-17 09:32:30,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] spmStart polling ended, spm status: SPM
>2012-12-17 09:32:30,767 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ab5ab054-c822-49ec-9ae0-62ec47da7210), log id: 3c693cd2
>2012-12-17 09:32:30,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, HSMClearTaskVDSCommand, log id: 3c693cd2
>2012-12-17 09:32:30,786 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@21181dfa, log id: 58ee0f0c
>2012-12-17 09:32:31,119 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:32:31,748 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [4f4f0222] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:32:31,748 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2695aa2a
>2012-12-17 09:32:31,815 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:32:32,210 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-60) [1c20be04] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:32:32,302 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [1c20be04] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 739259fd
>2012-12-17 09:32:32,303 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [1c20be04] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4c33f3bd
>2012-12-17 09:32:32,323 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [1c20be04] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:32,661 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [1c20be04] FINISH, SpmStopVDSCommand, log id: 4c33f3bd
>2012-12-17 09:32:32,675 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [1c20be04] FINISH, ResetIrsVDSCommand, log id: 739259fd
>2012-12-17 09:32:32,686 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [1c20be04] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 66f45f6d
>2012-12-17 09:32:32,686 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [1c20be04] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:32,686 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [1c20be04] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 66f45f6d
>2012-12-17 09:32:32,757 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4f4f0222] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:32:32,769 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4f4f0222] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4886
>2012-12-17 09:32:32,771 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4886, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1bd1b2cc
>2012-12-17 09:32:32,793 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] spmStart polling started: taskId = 461bae4f-8e19-444d-8a04-60e4e10b8565
>2012-12-17 09:32:33,825 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] spmStart polling ended: taskId = 461bae4f-8e19-444d-8a04-60e4e10b8565 task status = finished
>2012-12-17 09:32:33,844 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] spmStart polling ended, spm status: SPM
>2012-12-17 09:32:33,846 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=461bae4f-8e19-444d-8a04-60e4e10b8565), log id: 3185d6e4
>2012-12-17 09:32:33,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, HSMClearTaskVDSCommand, log id: 3185d6e4
>2012-12-17 09:32:33,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@49df2c22, log id: 1bd1b2cc
>2012-12-17 09:32:33,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4f4f0222] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:32:33,923 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:33,925 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 11adbdc5
>2012-12-17 09:32:33,946 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 11adbdc5
>2012-12-17 09:32:33,946 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2695aa2a
>2012-12-17 09:32:33,946 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [4f4f0222] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:32:33,946 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4cf4e495
>2012-12-17 09:32:33,946 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:33,948 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6251276c
>2012-12-17 09:32:33,966 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6251276c
>2012-12-17 09:32:33,966 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4cf4e495
>2012-12-17 09:32:33,966 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [4f4f0222] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:32:35,010 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:32:42,825 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [13d7977] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:32:42,926 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [4ea30a59] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:32:43,006 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4131f693
>2012-12-17 09:32:43,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 64ba3c8e
>2012-12-17 09:32:43,028 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:43,194 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, SpmStopVDSCommand, log id: 64ba3c8e
>2012-12-17 09:32:43,225 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, ResetIrsVDSCommand, log id: 4131f693
>2012-12-17 09:32:43,236 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 10111302
>2012-12-17 09:32:43,236 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:43,236 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 10111302
>2012-12-17 09:32:45,226 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [23d69744] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:32:53,323 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [6e561ed4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:32:53,352 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [6e561ed4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4887
>2012-12-17 09:32:53,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4887, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6bef30a
>2012-12-17 09:32:53,380 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] spmStart polling started: taskId = 64cae0d6-9741-4e03-97a0-fdd2d89de04c
>2012-12-17 09:32:54,401 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] spmStart polling ended: taskId = 64cae0d6-9741-4e03-97a0-fdd2d89de04c task status = finished
>2012-12-17 09:32:54,419 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] spmStart polling ended, spm status: SPM
>2012-12-17 09:32:54,421 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=64cae0d6-9741-4e03-97a0-fdd2d89de04c), log id: 5efc183d
>2012-12-17 09:32:54,438 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] FINISH, HSMClearTaskVDSCommand, log id: 5efc183d
>2012-12-17 09:32:54,439 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@789b7d82, log id: 6bef30a
>2012-12-17 09:32:54,475 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [6e561ed4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:32:54,532 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [651c3e82] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:32:54,533 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6eed47f6
>2012-12-17 09:32:54,562 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [6e561ed4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:32:54,739 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [5bc13aa6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:32:54,839 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [5bc13aa6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2ab6a0da
>2012-12-17 09:32:54,841 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [5bc13aa6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: a1d8b11
>2012-12-17 09:32:54,862 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [5bc13aa6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:55,013 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [5bc13aa6] FINISH, SpmStopVDSCommand, log id: a1d8b11
>2012-12-17 09:32:55,045 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [5bc13aa6] FINISH, ResetIrsVDSCommand, log id: 2ab6a0da
>2012-12-17 09:32:55,056 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [5bc13aa6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 597c072c
>2012-12-17 09:32:55,056 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [5bc13aa6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:55,056 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [5bc13aa6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 597c072c
>2012-12-17 09:32:55,135 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [651c3e82] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:32:55,160 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [651c3e82] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4888
>2012-12-17 09:32:55,161 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4888, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6934a32
>2012-12-17 09:32:55,185 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] spmStart polling started: taskId = 1dc37269-2d6b-4e27-95a6-5a08bab7e191
>2012-12-17 09:32:56,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] spmStart polling ended: taskId = 1dc37269-2d6b-4e27-95a6-5a08bab7e191 task status = finished
>2012-12-17 09:32:56,224 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] spmStart polling ended, spm status: SPM
>2012-12-17 09:32:56,225 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1dc37269-2d6b-4e27-95a6-5a08bab7e191), log id: f30b73b
>2012-12-17 09:32:56,243 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, HSMClearTaskVDSCommand, log id: f30b73b
>2012-12-17 09:32:56,243 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@474d8a47, log id: 6934a32
>2012-12-17 09:32:56,333 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [651c3e82] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:32:56,367 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:56,372 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4b53ce38
>2012-12-17 09:32:56,393 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b53ce38
>2012-12-17 09:32:56,394 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6eed47f6
>2012-12-17 09:32:56,394 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [651c3e82] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:32:56,394 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6b07e52a
>2012-12-17 09:32:56,394 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:32:56,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 65bc4240
>2012-12-17 09:32:56,415 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 65bc4240
>2012-12-17 09:32:56,415 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6b07e52a
>2012-12-17 09:32:56,415 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [651c3e82] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:33:05,189 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [15f75f8d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:33:05,306 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [519a0bb9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:33:05,407 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7766e646
>2012-12-17 09:33:05,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 104469f8
>2012-12-17 09:33:05,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:33:06,077 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] FINISH, SpmStopVDSCommand, log id: 104469f8
>2012-12-17 09:33:06,570 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] FINISH, ResetIrsVDSCommand, log id: 7766e646
>2012-12-17 09:33:06,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6647da41
>2012-12-17 09:33:06,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [519a0bb9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:33:06,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6647da41
>2012-12-17 09:33:06,677 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:33:16,882 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [3cb7ad26] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:33:16,906 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [3cb7ad26] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4889
>2012-12-17 09:33:16,965 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4889, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4c485ad9
>2012-12-17 09:33:16,990 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] spmStart polling started: taskId = 3e13efc3-22e9-45fa-9907-ca4183ba9c33
>2012-12-17 09:33:18,012 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] spmStart polling ended: taskId = 3e13efc3-22e9-45fa-9907-ca4183ba9c33 task status = finished
>2012-12-17 09:33:18,031 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] spmStart polling ended, spm status: SPM
>2012-12-17 09:33:18,032 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3e13efc3-22e9-45fa-9907-ca4183ba9c33), log id: 5640d60
>2012-12-17 09:33:18,050 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] FINISH, HSMClearTaskVDSCommand, log id: 5640d60
>2012-12-17 09:33:18,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1dac3d52, log id: 4c485ad9
>2012-12-17 09:33:18,063 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [3cb7ad26] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:33:18,106 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [263a433b] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:33:18,106 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2281b1de
>2012-12-17 09:33:18,120 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [3cb7ad26] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:33:18,219 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [632afe2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:33:18,294 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [632afe2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 655fa7db
>2012-12-17 09:33:18,296 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [632afe2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 20fffb91
>2012-12-17 09:33:18,316 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [632afe2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:33:18,456 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [632afe2] FINISH, SpmStopVDSCommand, log id: 20fffb91
>2012-12-17 09:33:18,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [632afe2] FINISH, ResetIrsVDSCommand, log id: 655fa7db
>2012-12-17 09:33:18,492 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [632afe2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 123b8401
>2012-12-17 09:33:18,492 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [632afe2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:33:18,492 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [632afe2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 123b8401
>2012-12-17 09:33:18,606 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:33:18,669 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4890
>2012-12-17 09:33:18,670 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4890, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 33359f37
>2012-12-17 09:33:18,693 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] spmStart polling started: taskId = f3ff19a0-e5f2-41b2-9bc5-5ea5cc3e0f0d
>2012-12-17 09:33:19,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] spmStart polling ended: taskId = f3ff19a0-e5f2-41b2-9bc5-5ea5cc3e0f0d task status = finished
>2012-12-17 09:33:19,735 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] spmStart polling ended, spm status: SPM
>2012-12-17 09:33:19,737 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f3ff19a0-e5f2-41b2-9bc5-5ea5cc3e0f0d), log id: 11b5bbfd
>2012-12-17 09:33:19,756 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, HSMClearTaskVDSCommand, log id: 11b5bbfd
>2012-12-17 09:33:19,756 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3248e820, log id: 33359f37
>2012-12-17 09:33:19,788 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:33:19,808 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:33:19,810 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 719c5552
>2012-12-17 09:33:19,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 719c5552
>2012-12-17 09:33:19,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2281b1de
>2012-12-17 09:33:19,830 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [263a433b] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:33:19,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4ea5b4c3
>2012-12-17 09:33:19,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:33:19,832 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2f2dbc14
>2012-12-17 09:33:19,852 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2f2dbc14
>2012-12-17 09:33:19,852 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4ea5b4c3
>2012-12-17 09:33:19,852 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [263a433b] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:33:27,246 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:33:28,662 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [d7157bb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:33:28,761 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-99) [20d09da7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:33:28,864 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6862fda1
>2012-12-17 09:33:28,866 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7df27e52
>2012-12-17 09:33:28,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:33:29,585 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, SpmStopVDSCommand, log id: 7df27e52
>2012-12-17 09:33:29,596 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, ResetIrsVDSCommand, log id: 6862fda1
>2012-12-17 09:33:29,607 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7aecc102
>2012-12-17 09:33:29,607 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [20d09da7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:33:29,607 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7aecc102
>2012-12-17 09:33:37,459 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:33:39,680 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:33:39,704 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4891
>2012-12-17 09:33:39,705 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4891, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 71cdfe23
>2012-12-17 09:33:39,727 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] spmStart polling started: taskId = 14253e40-f1a1-4353-95b4-8c38536e0571
>2012-12-17 09:33:40,751 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] spmStart polling ended: taskId = 14253e40-f1a1-4353-95b4-8c38536e0571 task status = finished
>2012-12-17 09:33:40,770 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] spmStart polling ended, spm status: SPM
>2012-12-17 09:33:40,772 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=14253e40-f1a1-4353-95b4-8c38536e0571), log id: 2f86007c
>2012-12-17 09:33:40,800 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, HSMClearTaskVDSCommand, log id: 2f86007c
>2012-12-17 09:33:40,800 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@576adb89, log id: 71cdfe23
>2012-12-17 09:33:40,882 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:33:41,025 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:33:41,025 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 72342b75
>2012-12-17 09:33:41,057 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:33:41,232 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-29) [680acadd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:33:41,309 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [680acadd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3ccd3485
>2012-12-17 09:33:41,310 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [680acadd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 13786ef0
>2012-12-17 09:33:41,330 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [680acadd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:33:41,473 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [680acadd] FINISH, SpmStopVDSCommand, log id: 13786ef0
>2012-12-17 09:33:41,485 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [680acadd] FINISH, ResetIrsVDSCommand, log id: 3ccd3485
>2012-12-17 09:33:41,496 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [680acadd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 114156fd
>2012-12-17 09:33:41,496 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [680acadd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:33:41,496 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [680acadd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 114156fd
>2012-12-17 09:33:41,582 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:33:41,600 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4892
>2012-12-17 09:33:41,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4892, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: bf76416
>2012-12-17 09:33:41,623 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling started: taskId = ab93a84a-8847-4966-9c0a-5828b2b4835c
>2012-12-17 09:33:42,646 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended: taskId = ab93a84a-8847-4966-9c0a-5828b2b4835c task status = finished
>2012-12-17 09:33:42,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended, spm status: SPM
>2012-12-17 09:33:42,665 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ab93a84a-8847-4966-9c0a-5828b2b4835c), log id: 6d5cddae
>2012-12-17 09:33:42,684 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMClearTaskVDSCommand, log id: 6d5cddae
>2012-12-17 09:33:42,684 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6bde0299, log id: bf76416
>2012-12-17 09:33:42,703 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:33:42,720 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:33:42,724 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1b89b64a
>2012-12-17 09:33:42,744 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1b89b64a
>2012-12-17 09:33:42,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 72342b75
>2012-12-17 09:33:42,744 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:33:42,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e1ec265
>2012-12-17 09:33:42,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:33:42,745 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6c0c8117
>2012-12-17 09:33:42,764 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6c0c8117
>2012-12-17 09:33:42,764 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e1ec265
>2012-12-17 09:33:42,764 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:33:47,657 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:33:51,624 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [68b3db6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:33:51,791 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [56a7c4a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:33:51,907 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7b975bfc
>2012-12-17 09:33:51,909 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 458c2f56
>2012-12-17 09:33:51,929 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:33:52,116 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, SpmStopVDSCommand, log id: 458c2f56
>2012-12-17 09:33:52,135 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, ResetIrsVDSCommand, log id: 7b975bfc
>2012-12-17 09:33:52,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44d3ce62
>2012-12-17 09:33:52,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:33:52,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44d3ce62
>2012-12-17 09:33:58,253 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6b59f114] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:34:02,220 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:34:02,249 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4893
>2012-12-17 09:34:02,250 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4893, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3decbd81
>2012-12-17 09:34:02,273 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] spmStart polling started: taskId = 4f2185bc-a034-4630-8db5-de6a45452d80
>2012-12-17 09:34:03,295 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] spmStart polling ended: taskId = 4f2185bc-a034-4630-8db5-de6a45452d80 task status = finished
>2012-12-17 09:34:03,313 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] spmStart polling ended, spm status: SPM
>2012-12-17 09:34:03,315 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4f2185bc-a034-4630-8db5-de6a45452d80), log id: 5d7f6966
>2012-12-17 09:34:03,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] FINISH, HSMClearTaskVDSCommand, log id: 5d7f6966
>2012-12-17 09:34:03,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@57e0d77e, log id: 3decbd81
>2012-12-17 09:34:03,346 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:34:03,394 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-61) [22fa8e65] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:34:03,394 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2bbd5a7c
>2012-12-17 09:34:03,405 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:34:03,814 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [248935dc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:34:03,911 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [248935dc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 28344361
>2012-12-17 09:34:03,912 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [248935dc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2348d3b0
>2012-12-17 09:34:03,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [248935dc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:04,064 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [248935dc] FINISH, SpmStopVDSCommand, log id: 2348d3b0
>2012-12-17 09:34:04,084 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [248935dc] FINISH, ResetIrsVDSCommand, log id: 28344361
>2012-12-17 09:34:04,096 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [248935dc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 53d6a1be
>2012-12-17 09:34:04,096 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [248935dc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:04,096 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [248935dc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 53d6a1be
>2012-12-17 09:34:04,156 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [22fa8e65] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:34:04,168 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [22fa8e65] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4894
>2012-12-17 09:34:04,169 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4894, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3fc15b4e
>2012-12-17 09:34:04,191 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] spmStart polling started: taskId = 2d6245f2-535f-4bef-9fe2-f63187688762
>2012-12-17 09:34:05,212 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] spmStart polling ended: taskId = 2d6245f2-535f-4bef-9fe2-f63187688762 task status = finished
>2012-12-17 09:34:05,233 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] spmStart polling ended, spm status: SPM
>2012-12-17 09:34:05,234 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2d6245f2-535f-4bef-9fe2-f63187688762), log id: 395a1f87
>2012-12-17 09:34:05,252 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, HSMClearTaskVDSCommand, log id: 395a1f87
>2012-12-17 09:34:05,252 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4bcbf57b, log id: 3fc15b4e
>2012-12-17 09:34:05,286 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [22fa8e65] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:34:05,341 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:05,343 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3c8604f9
>2012-12-17 09:34:05,363 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3c8604f9
>2012-12-17 09:34:05,363 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2bbd5a7c
>2012-12-17 09:34:05,363 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-61) [22fa8e65] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:34:05,363 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2b0967a
>2012-12-17 09:34:05,363 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:05,365 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 598c391a
>2012-12-17 09:34:05,383 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 598c391a
>2012-12-17 09:34:05,383 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2b0967a
>2012-12-17 09:34:05,383 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-61) [22fa8e65] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:34:08,447 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [e8edb34] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:34:14,260 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [20c0de80] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:34:14,464 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [3c4cd577] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:34:14,557 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 36fd4c2f
>2012-12-17 09:34:14,558 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 21e7e53f
>2012-12-17 09:34:14,577 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:14,778 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] FINISH, SpmStopVDSCommand, log id: 21e7e53f
>2012-12-17 09:34:14,801 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] FINISH, ResetIrsVDSCommand, log id: 36fd4c2f
>2012-12-17 09:34:14,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5daa386a
>2012-12-17 09:34:14,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3c4cd577] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:14,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5daa386a
>2012-12-17 09:34:18,666 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:34:24,877 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:34:24,913 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4895
>2012-12-17 09:34:24,915 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4895, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2c40109b
>2012-12-17 09:34:24,937 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling started: taskId = 5d7fbd7e-29e2-4420-89db-ff1ad972f0ea
>2012-12-17 09:34:25,960 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling ended: taskId = 5d7fbd7e-29e2-4420-89db-ff1ad972f0ea task status = finished
>2012-12-17 09:34:25,979 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling ended, spm status: SPM
>2012-12-17 09:34:25,980 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5d7fbd7e-29e2-4420-89db-ff1ad972f0ea), log id: 4f3ba92
>2012-12-17 09:34:25,998 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, HSMClearTaskVDSCommand, log id: 4f3ba92
>2012-12-17 09:34:25,999 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1d5dc9c8, log id: 2c40109b
>2012-12-17 09:34:26,024 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:34:26,291 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:34:26,291 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5fa88926
>2012-12-17 09:34:26,313 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:34:26,497 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-33) [47513fa0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:34:26,598 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 40dde40
>2012-12-17 09:34:26,652 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 307b6232
>2012-12-17 09:34:26,673 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:26,803 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, SpmStopVDSCommand, log id: 307b6232
>2012-12-17 09:34:26,850 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, ResetIrsVDSCommand, log id: 40dde40
>2012-12-17 09:34:26,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5304d3ef
>2012-12-17 09:34:26,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:26,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5304d3ef
>2012-12-17 09:34:27,016 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:34:27,034 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4896
>2012-12-17 09:34:27,036 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4896, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6cea505f
>2012-12-17 09:34:27,059 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling started: taskId = d7741f8e-a3cc-4230-9239-5e3dcf1b4ab7
>2012-12-17 09:34:28,082 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended: taskId = d7741f8e-a3cc-4230-9239-5e3dcf1b4ab7 task status = finished
>2012-12-17 09:34:28,100 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended, spm status: SPM
>2012-12-17 09:34:28,102 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d7741f8e-a3cc-4230-9239-5e3dcf1b4ab7), log id: ce92b62
>2012-12-17 09:34:28,120 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMClearTaskVDSCommand, log id: ce92b62
>2012-12-17 09:34:28,120 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@201f358d, log id: 6cea505f
>2012-12-17 09:34:28,142 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:34:28,162 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:28,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3b868cd8
>2012-12-17 09:34:28,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3b868cd8
>2012-12-17 09:34:28,187 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5fa88926
>2012-12-17 09:34:28,187 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:34:28,187 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 32b2652d
>2012-12-17 09:34:28,188 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:28,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5baf1a4d
>2012-12-17 09:34:28,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5baf1a4d
>2012-12-17 09:34:28,208 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 32b2652d
>2012-12-17 09:34:28,208 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:34:28,847 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [680acadd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:34:38,355 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [5f63eb29] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:34:38,692 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-81) [5117cf2d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:34:38,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7f8d3f70
>2012-12-17 09:34:38,767 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 46ecaa03
>2012-12-17 09:34:38,787 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:38,938 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, SpmStopVDSCommand, log id: 46ecaa03
>2012-12-17 09:34:38,952 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, ResetIrsVDSCommand, log id: 7f8d3f70
>2012-12-17 09:34:38,963 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7593f22a
>2012-12-17 09:34:38,963 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [5117cf2d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:38,963 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7593f22a
>2012-12-17 09:34:41,285 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:34:49,026 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6de308ea] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:34:49,047 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6de308ea] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4897
>2012-12-17 09:34:49,048 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4897, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e983992
>2012-12-17 09:34:49,070 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] spmStart polling started: taskId = 86c07b01-3a81-477f-84bd-8e0559cb534b
>2012-12-17 09:34:50,093 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] spmStart polling ended: taskId = 86c07b01-3a81-477f-84bd-8e0559cb534b task status = finished
>2012-12-17 09:34:50,112 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] spmStart polling ended, spm status: SPM
>2012-12-17 09:34:50,113 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=86c07b01-3a81-477f-84bd-8e0559cb534b), log id: 37931e7c
>2012-12-17 09:34:50,131 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] FINISH, HSMClearTaskVDSCommand, log id: 37931e7c
>2012-12-17 09:34:50,131 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1bb67b65, log id: 3e983992
>2012-12-17 09:34:50,142 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6de308ea] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:34:50,177 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:34:50,177 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2cf64ece
>2012-12-17 09:34:50,197 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6de308ea] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:34:50,294 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-56) [6728cccd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:34:50,338 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [6728cccd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 48888843
>2012-12-17 09:34:50,339 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [6728cccd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4274287
>2012-12-17 09:34:50,359 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [6728cccd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:50,733 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [6728cccd] FINISH, SpmStopVDSCommand, log id: 4274287
>2012-12-17 09:34:51,318 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [6728cccd] FINISH, ResetIrsVDSCommand, log id: 48888843
>2012-12-17 09:34:51,329 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [6728cccd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6f35b6cd
>2012-12-17 09:34:51,329 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6728cccd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:51,329 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [6728cccd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6f35b6cd
>2012-12-17 09:34:51,384 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:34:51,408 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4898
>2012-12-17 09:34:51,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4898, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 504f3eae
>2012-12-17 09:34:51,431 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling started: taskId = da3e029a-8e98-4c5c-8b6b-cd39190c533d
>2012-12-17 09:34:52,453 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling ended: taskId = da3e029a-8e98-4c5c-8b6b-cd39190c533d task status = finished
>2012-12-17 09:34:52,472 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling ended, spm status: SPM
>2012-12-17 09:34:52,473 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=da3e029a-8e98-4c5c-8b6b-cd39190c533d), log id: 579277e3
>2012-12-17 09:34:52,491 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMClearTaskVDSCommand, log id: 579277e3
>2012-12-17 09:34:52,491 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@50fd04ff, log id: 504f3eae
>2012-12-17 09:34:52,502 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:34:52,514 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:52,517 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2055255
>2012-12-17 09:34:52,538 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2055255
>2012-12-17 09:34:52,538 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2cf64ece
>2012-12-17 09:34:52,538 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:34:52,538 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3efbd023
>2012-12-17 09:34:52,538 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:34:52,540 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6694de54
>2012-12-17 09:34:52,558 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6694de54
>2012-12-17 09:34:52,558 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3efbd023
>2012-12-17 09:34:52,558 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:35:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [6e84ec96] Checking autorecoverable hosts
>2012-12-17 09:35:00,012 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [6e84ec96] Autorecovering 0 hosts
>2012-12-17 09:35:00,013 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [6e84ec96] Checking autorecoverable hosts done
>2012-12-17 09:35:00,013 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [6e84ec96] Autorecovering storage domains is disabled, skipping
>2012-12-17 09:35:01,464 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [e42ef5e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:35:01,608 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-15) [4f008eee] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:35:01,692 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6c694e10
>2012-12-17 09:35:01,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: adb73d0
>2012-12-17 09:35:01,727 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:01,884 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] FINISH, SpmStopVDSCommand, log id: adb73d0
>2012-12-17 09:35:01,895 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] FINISH, ResetIrsVDSCommand, log id: 6c694e10
>2012-12-17 09:35:01,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4b031d7e
>2012-12-17 09:35:01,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [4f008eee] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:01,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4b031d7e
>2012-12-17 09:35:02,041 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:35:11,985 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:35:12,041 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4899
>2012-12-17 09:35:12,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4899, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 297ebee2
>2012-12-17 09:35:12,064 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] spmStart polling started: taskId = c6121edf-5dc4-49c8-8161-87a076b4c9ad
>2012-12-17 09:35:13,085 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] spmStart polling ended: taskId = c6121edf-5dc4-49c8-8161-87a076b4c9ad task status = finished
>2012-12-17 09:35:13,103 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] spmStart polling ended, spm status: SPM
>2012-12-17 09:35:13,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c6121edf-5dc4-49c8-8161-87a076b4c9ad), log id: 3950effd
>2012-12-17 09:35:13,122 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, HSMClearTaskVDSCommand, log id: 3950effd
>2012-12-17 09:35:13,123 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@70bbe870, log id: 297ebee2
>2012-12-17 09:35:13,209 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:35:13,252 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [47513fa0] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:35:13,252 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1aa90823
>2012-12-17 09:35:13,274 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:35:13,392 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [24a780e1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:35:13,440 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [24a780e1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1ea6a0e8
>2012-12-17 09:35:13,441 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [24a780e1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3595ca6
>2012-12-17 09:35:13,460 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [24a780e1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:13,603 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [24a780e1] FINISH, SpmStopVDSCommand, log id: 3595ca6
>2012-12-17 09:35:13,619 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [24a780e1] FINISH, ResetIrsVDSCommand, log id: 1ea6a0e8
>2012-12-17 09:35:13,629 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [24a780e1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 36c7e5f0
>2012-12-17 09:35:13,629 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [24a780e1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:13,629 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [24a780e1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 36c7e5f0
>2012-12-17 09:35:13,681 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:35:13,695 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4900
>2012-12-17 09:35:13,696 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4900, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4ee5cd08
>2012-12-17 09:35:13,718 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] spmStart polling started: taskId = 8a34f209-108f-4245-9cce-78366d415c39
>2012-12-17 09:35:14,739 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] spmStart polling ended: taskId = 8a34f209-108f-4245-9cce-78366d415c39 task status = finished
>2012-12-17 09:35:14,758 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] spmStart polling ended, spm status: SPM
>2012-12-17 09:35:14,759 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8a34f209-108f-4245-9cce-78366d415c39), log id: 5a7eb679
>2012-12-17 09:35:14,777 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, HSMClearTaskVDSCommand, log id: 5a7eb679
>2012-12-17 09:35:14,777 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@34fdf059, log id: 4ee5cd08
>2012-12-17 09:35:14,794 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:35:14,834 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:14,835 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 8407678
>2012-12-17 09:35:14,857 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 8407678
>2012-12-17 09:35:14,857 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1aa90823
>2012-12-17 09:35:14,857 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [47513fa0] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:35:14,857 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 68ec9559
>2012-12-17 09:35:14,857 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:14,858 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4311d6a1
>2012-12-17 09:35:14,877 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4311d6a1
>2012-12-17 09:35:14,877 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 68ec9559
>2012-12-17 09:35:14,877 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [47513fa0] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:35:22,399 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [76c72cb4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:35:23,722 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [26c9f399] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:35:23,805 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [588a5ec3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:35:23,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5a77ac42
>2012-12-17 09:35:23,863 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6b89cd7b
>2012-12-17 09:35:23,883 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:24,056 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] FINISH, SpmStopVDSCommand, log id: 6b89cd7b
>2012-12-17 09:35:24,069 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] FINISH, ResetIrsVDSCommand, log id: 5a77ac42
>2012-12-17 09:35:24,080 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3c897660
>2012-12-17 09:35:24,080 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [588a5ec3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:24,080 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3c897660
>2012-12-17 09:35:32,573 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [134ba83e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:35:34,130 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6b59f114] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:35:34,147 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6b59f114] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4901
>2012-12-17 09:35:34,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4901, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5595c69a
>2012-12-17 09:35:34,172 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] spmStart polling started: taskId = 0c5ffeea-aa83-479f-850d-8636d2e6bc5a
>2012-12-17 09:35:36,218 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] spmStart polling ended: taskId = 0c5ffeea-aa83-479f-850d-8636d2e6bc5a task status = finished
>2012-12-17 09:35:36,237 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] spmStart polling ended, spm status: SPM
>2012-12-17 09:35:36,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0c5ffeea-aa83-479f-850d-8636d2e6bc5a), log id: 39e22398
>2012-12-17 09:35:36,257 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] FINISH, HSMClearTaskVDSCommand, log id: 39e22398
>2012-12-17 09:35:36,257 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2d8eedd8, log id: 5595c69a
>2012-12-17 09:35:36,276 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6b59f114] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:35:36,385 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [6c899ac8] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:35:36,385 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4c57853c
>2012-12-17 09:35:36,433 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6b59f114] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:35:36,626 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [3623f38c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:35:36,686 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [3623f38c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 79a88e45
>2012-12-17 09:35:36,687 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [3623f38c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 11174adf
>2012-12-17 09:35:36,707 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [3623f38c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:37,258 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [3623f38c] FINISH, SpmStopVDSCommand, log id: 11174adf
>2012-12-17 09:35:37,272 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [3623f38c] FINISH, ResetIrsVDSCommand, log id: 79a88e45
>2012-12-17 09:35:37,283 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [3623f38c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2e63c62e
>2012-12-17 09:35:37,283 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [3623f38c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:37,283 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [3623f38c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2e63c62e
>2012-12-17 09:35:37,333 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:35:37,349 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4902
>2012-12-17 09:35:37,350 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4902, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 106db34b
>2012-12-17 09:35:37,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] spmStart polling started: taskId = 37554cd1-fddc-4a0f-abba-b722833109f8
>2012-12-17 09:35:38,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] spmStart polling ended: taskId = 37554cd1-fddc-4a0f-abba-b722833109f8 task status = finished
>2012-12-17 09:35:38,418 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] spmStart polling ended, spm status: SPM
>2012-12-17 09:35:38,419 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=37554cd1-fddc-4a0f-abba-b722833109f8), log id: 65e9ae6a
>2012-12-17 09:35:38,438 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, HSMClearTaskVDSCommand, log id: 65e9ae6a
>2012-12-17 09:35:38,438 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@50b9e56b, log id: 106db34b
>2012-12-17 09:35:38,466 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:35:38,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:38,486 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 32a2a052
>2012-12-17 09:35:38,505 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 32a2a052
>2012-12-17 09:35:38,505 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4c57853c
>2012-12-17 09:35:38,505 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [6c899ac8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:35:38,505 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 52cb8176
>2012-12-17 09:35:38,505 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:38,506 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7e20938f
>2012-12-17 09:35:38,525 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7e20938f
>2012-12-17 09:35:38,525 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 52cb8176
>2012-12-17 09:35:38,525 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [6c899ac8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:35:42,802 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [78e614fb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:35:47,372 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [1813139a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:35:47,466 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [134cc8af] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:35:47,513 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3e35d1b
>2012-12-17 09:35:47,514 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4a65acc3
>2012-12-17 09:35:47,534 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:47,710 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] FINISH, SpmStopVDSCommand, log id: 4a65acc3
>2012-12-17 09:35:47,723 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] FINISH, ResetIrsVDSCommand, log id: 3e35d1b
>2012-12-17 09:35:47,734 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 41f155f5
>2012-12-17 09:35:47,734 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [134cc8af] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:47,734 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 41f155f5
>2012-12-17 09:35:52,984 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:35:57,785 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [23d69744] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:35:57,794 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [23d69744] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4903
>2012-12-17 09:35:57,796 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [23d69744] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4903, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 27f340d7
>2012-12-17 09:35:57,819 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [23d69744] spmStart polling started: taskId = 1fa20210-8269-4a07-bed2-a716fd5ea40d
>2012-12-17 09:35:58,842 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [23d69744] spmStart polling ended: taskId = 1fa20210-8269-4a07-bed2-a716fd5ea40d task status = finished
>2012-12-17 09:35:58,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [23d69744] spmStart polling ended, spm status: SPM
>2012-12-17 09:35:58,861 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [23d69744] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1fa20210-8269-4a07-bed2-a716fd5ea40d), log id: 54e5eb19
>2012-12-17 09:35:58,879 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [23d69744] FINISH, HSMClearTaskVDSCommand, log id: 54e5eb19
>2012-12-17 09:35:58,879 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [23d69744] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42ade1da, log id: 27f340d7
>2012-12-17 09:35:58,891 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [23d69744] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:35:59,018 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [6157bb5e] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:35:59,018 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 361355fc
>2012-12-17 09:35:59,031 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [23d69744] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:35:59,184 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [6f4b2c75] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:35:59,240 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5eaaedb4
>2012-12-17 09:35:59,242 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5ae77271
>2012-12-17 09:35:59,261 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:59,393 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] FINISH, SpmStopVDSCommand, log id: 5ae77271
>2012-12-17 09:35:59,408 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] FINISH, ResetIrsVDSCommand, log id: 5eaaedb4
>2012-12-17 09:35:59,419 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 521d9b6d
>2012-12-17 09:35:59,419 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6f4b2c75] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:35:59,419 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 521d9b6d
>2012-12-17 09:35:59,491 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [6157bb5e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:35:59,508 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [6157bb5e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4904
>2012-12-17 09:35:59,510 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4904, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 413510a2
>2012-12-17 09:35:59,536 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] spmStart polling started: taskId = bf526500-b163-4b84-8888-728a0730dc8a
>2012-12-17 09:36:00,558 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] spmStart polling ended: taskId = bf526500-b163-4b84-8888-728a0730dc8a task status = finished
>2012-12-17 09:36:00,577 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] spmStart polling ended, spm status: SPM
>2012-12-17 09:36:00,578 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bf526500-b163-4b84-8888-728a0730dc8a), log id: c38863f
>2012-12-17 09:36:00,596 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, HSMClearTaskVDSCommand, log id: c38863f
>2012-12-17 09:36:00,596 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@52f5985d, log id: 413510a2
>2012-12-17 09:36:01,059 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [6157bb5e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:36:01,178 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:01,183 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: c4ef89e
>2012-12-17 09:36:01,203 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: c4ef89e
>2012-12-17 09:36:01,203 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 361355fc
>2012-12-17 09:36:01,203 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [6157bb5e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:36:01,203 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 14a45362
>2012-12-17 09:36:01,203 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:01,204 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4033a5c9
>2012-12-17 09:36:01,223 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4033a5c9
>2012-12-17 09:36:01,223 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 14a45362
>2012-12-17 09:36:01,223 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [6157bb5e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:36:03,191 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [63fee8f1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:36:09,531 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3fe36f5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:36:09,657 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-76) [46833f1f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:36:09,694 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4a9d735f
>2012-12-17 09:36:09,696 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7d1785f
>2012-12-17 09:36:09,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:09,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] FINISH, SpmStopVDSCommand, log id: 7d1785f
>2012-12-17 09:36:09,867 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] FINISH, ResetIrsVDSCommand, log id: 4a9d735f
>2012-12-17 09:36:09,878 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7c76af09
>2012-12-17 09:36:09,878 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [46833f1f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:09,878 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7c76af09
>2012-12-17 09:36:13,367 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:36:19,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:36:20,044 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4905
>2012-12-17 09:36:20,046 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4905, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6ddece82
>2012-12-17 09:36:20,069 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling started: taskId = b55918ca-9ce6-44f7-adc0-15082b41077e
>2012-12-17 09:36:21,092 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling ended: taskId = b55918ca-9ce6-44f7-adc0-15082b41077e task status = finished
>2012-12-17 09:36:21,119 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling ended, spm status: SPM
>2012-12-17 09:36:21,120 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b55918ca-9ce6-44f7-adc0-15082b41077e), log id: 5a0eded5
>2012-12-17 09:36:21,139 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, HSMClearTaskVDSCommand, log id: 5a0eded5
>2012-12-17 09:36:21,139 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@647a024, log id: 6ddece82
>2012-12-17 09:36:21,901 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:36:22,030 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [3a1c2706] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:36:22,031 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6bde588
>2012-12-17 09:36:22,079 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:36:22,284 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-58) [3ad3ad88] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:36:22,327 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 39bc936
>2012-12-17 09:36:22,329 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1b1b73ac
>2012-12-17 09:36:22,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:22,468 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] FINISH, SpmStopVDSCommand, log id: 1b1b73ac
>2012-12-17 09:36:22,489 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] FINISH, ResetIrsVDSCommand, log id: 39bc936
>2012-12-17 09:36:22,500 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 35e1049a
>2012-12-17 09:36:22,500 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [3ad3ad88] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:22,500 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 35e1049a
>2012-12-17 09:36:22,538 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:36:22,550 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4906
>2012-12-17 09:36:22,551 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4906, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 515c06da
>2012-12-17 09:36:22,573 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] spmStart polling started: taskId = a6eacabb-6e2b-477f-95de-3df25a85d5e1
>2012-12-17 09:36:23,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] spmStart polling ended: taskId = a6eacabb-6e2b-477f-95de-3df25a85d5e1 task status = finished
>2012-12-17 09:36:23,613 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] spmStart polling ended, spm status: SPM
>2012-12-17 09:36:23,614 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a6eacabb-6e2b-477f-95de-3df25a85d5e1), log id: 1a1e6aaf
>2012-12-17 09:36:23,633 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, HSMClearTaskVDSCommand, log id: 1a1e6aaf
>2012-12-17 09:36:23,633 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42ee148e, log id: 515c06da
>2012-12-17 09:36:23,646 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:36:23,658 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:23,660 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4e8be767
>2012-12-17 09:36:23,679 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4e8be767
>2012-12-17 09:36:23,679 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6bde588
>2012-12-17 09:36:23,679 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [3a1c2706] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:36:23,680 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: e9120f6
>2012-12-17 09:36:23,680 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:23,681 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2867c319
>2012-12-17 09:36:23,699 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2867c319
>2012-12-17 09:36:23,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: e9120f6
>2012-12-17 09:36:23,699 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [3a1c2706] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:36:32,589 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:36:32,738 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-37) [9b1d86] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:36:32,790 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4150d8c2
>2012-12-17 09:36:32,795 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 757266b8
>2012-12-17 09:36:32,825 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:33,257 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] FINISH, SpmStopVDSCommand, log id: 757266b8
>2012-12-17 09:36:33,297 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] FINISH, ResetIrsVDSCommand, log id: 4150d8c2
>2012-12-17 09:36:33,309 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 14fbde2
>2012-12-17 09:36:33,309 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [9b1d86] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:33,309 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 14fbde2
>2012-12-17 09:36:33,738 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:36:43,366 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [47111e6d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:36:43,380 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [47111e6d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4907
>2012-12-17 09:36:43,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4907, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 58c1ba2f
>2012-12-17 09:36:43,402 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] spmStart polling started: taskId = a021c2db-fa67-4743-8467-9eb36e724145
>2012-12-17 09:36:44,425 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] spmStart polling ended: taskId = a021c2db-fa67-4743-8467-9eb36e724145 task status = finished
>2012-12-17 09:36:44,444 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] spmStart polling ended, spm status: SPM
>2012-12-17 09:36:44,445 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a021c2db-fa67-4743-8467-9eb36e724145), log id: 77b95030
>2012-12-17 09:36:44,464 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] FINISH, HSMClearTaskVDSCommand, log id: 77b95030
>2012-12-17 09:36:44,464 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@273fbcb8, log id: 58c1ba2f
>2012-12-17 09:36:44,516 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [47111e6d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:36:44,577 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [7db92791] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:36:44,577 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1c5701b7
>2012-12-17 09:36:44,602 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [47111e6d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:36:44,765 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [515f648d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:36:44,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5ea5558a
>2012-12-17 09:36:44,835 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 69f7940b
>2012-12-17 09:36:44,855 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [515f648d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:44,983 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, SpmStopVDSCommand, log id: 69f7940b
>2012-12-17 09:36:44,999 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, ResetIrsVDSCommand, log id: 5ea5558a
>2012-12-17 09:36:45,011 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44bcbd15
>2012-12-17 09:36:45,011 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [515f648d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:45,011 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44bcbd15
>2012-12-17 09:36:45,067 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7db92791] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:36:45,078 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7db92791] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4908
>2012-12-17 09:36:45,080 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4908, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4cbb6160
>2012-12-17 09:36:45,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7db92791] spmStart polling started: taskId = 3c84b1d1-004b-4f00-b887-1575c8e1ac6a
>2012-12-17 09:36:46,126 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7db92791] spmStart polling ended: taskId = 3c84b1d1-004b-4f00-b887-1575c8e1ac6a task status = finished
>2012-12-17 09:36:46,146 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7db92791] spmStart polling ended, spm status: SPM
>2012-12-17 09:36:46,147 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3c84b1d1-004b-4f00-b887-1575c8e1ac6a), log id: 2ffc14d3
>2012-12-17 09:36:46,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, HSMClearTaskVDSCommand, log id: 2ffc14d3
>2012-12-17 09:36:46,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4ebb4141, log id: 4cbb6160
>2012-12-17 09:36:46,197 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7db92791] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:36:46,232 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:46,234 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3f18529e
>2012-12-17 09:36:46,255 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3f18529e
>2012-12-17 09:36:46,255 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1c5701b7
>2012-12-17 09:36:46,256 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [7db92791] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:36:46,256 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 55d8236e
>2012-12-17 09:36:46,256 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:46,257 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: e673a8c
>2012-12-17 09:36:46,277 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: e673a8c
>2012-12-17 09:36:46,277 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 55d8236e
>2012-12-17 09:36:46,277 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [7db92791] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:36:54,151 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5d1b6c83] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:36:55,117 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [651c3e82] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:36:55,234 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [681f9614] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:36:55,291 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [681f9614] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 195798c7
>2012-12-17 09:36:55,293 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [681f9614] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 42bf564f
>2012-12-17 09:36:55,313 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [681f9614] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:55,469 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [681f9614] FINISH, SpmStopVDSCommand, log id: 42bf564f
>2012-12-17 09:36:55,591 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [681f9614] FINISH, ResetIrsVDSCommand, log id: 195798c7
>2012-12-17 09:36:55,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [681f9614] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 45c354e2
>2012-12-17 09:36:55,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [681f9614] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:36:55,602 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [681f9614] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 45c354e2
>2012-12-17 09:37:04,333 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:37:06,150 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [570659e5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:37:06,728 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [570659e5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4909
>2012-12-17 09:37:06,729 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [570659e5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4909, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 77e97861
>2012-12-17 09:37:06,753 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [570659e5] spmStart polling started: taskId = 1b4541eb-d7a6-498a-a8bf-076a05734411
>2012-12-17 09:37:07,776 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [570659e5] spmStart polling ended: taskId = 1b4541eb-d7a6-498a-a8bf-076a05734411 task status = finished
>2012-12-17 09:37:07,794 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [570659e5] spmStart polling ended, spm status: SPM
>2012-12-17 09:37:07,796 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [570659e5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1b4541eb-d7a6-498a-a8bf-076a05734411), log id: 21be2b44
>2012-12-17 09:37:07,814 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [570659e5] FINISH, HSMClearTaskVDSCommand, log id: 21be2b44
>2012-12-17 09:37:07,814 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [570659e5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4c0a8eaa, log id: 77e97861
>2012-12-17 09:37:07,829 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [570659e5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:37:07,867 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:37:07,867 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1ae91126
>2012-12-17 09:37:07,891 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [570659e5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:37:08,003 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-3) [7646beaa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:37:08,041 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6da4d875
>2012-12-17 09:37:08,043 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7fdcabe3
>2012-12-17 09:37:08,062 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:08,198 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, SpmStopVDSCommand, log id: 7fdcabe3
>2012-12-17 09:37:08,216 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, ResetIrsVDSCommand, log id: 6da4d875
>2012-12-17 09:37:08,226 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 218cc4b1
>2012-12-17 09:37:08,226 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:08,226 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 218cc4b1
>2012-12-17 09:37:08,273 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:37:08,285 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4910
>2012-12-17 09:37:08,287 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4910, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 33afb4a4
>2012-12-17 09:37:08,308 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling started: taskId = f090005a-716f-4c55-a81d-0bf7e3e2c3f4
>2012-12-17 09:37:09,330 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling ended: taskId = f090005a-716f-4c55-a81d-0bf7e3e2c3f4 task status = finished
>2012-12-17 09:37:09,347 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling ended, spm status: SPM
>2012-12-17 09:37:09,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f090005a-716f-4c55-a81d-0bf7e3e2c3f4), log id: 26c8ea02
>2012-12-17 09:37:09,367 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMClearTaskVDSCommand, log id: 26c8ea02
>2012-12-17 09:37:09,367 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@507b3d32, log id: 33afb4a4
>2012-12-17 09:37:09,379 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:37:09,417 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:09,422 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 47d52282
>2012-12-17 09:37:09,440 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 47d52282
>2012-12-17 09:37:09,440 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1ae91126
>2012-12-17 09:37:09,440 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:37:09,441 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 79cca9a5
>2012-12-17 09:37:09,441 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:09,442 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7ebc8b16
>2012-12-17 09:37:09,461 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7ebc8b16
>2012-12-17 09:37:09,461 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 79cca9a5
>2012-12-17 09:37:09,461 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:37:14,519 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:37:18,314 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [1c20be04] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:37:18,450 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-60) [6eb2e264] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:37:18,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 663fa7c
>2012-12-17 09:37:18,583 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 11b77f47
>2012-12-17 09:37:18,602 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:18,752 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, SpmStopVDSCommand, log id: 11b77f47
>2012-12-17 09:37:18,764 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, ResetIrsVDSCommand, log id: 663fa7c
>2012-12-17 09:37:18,775 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 67adc956
>2012-12-17 09:37:18,775 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:18,775 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 67adc956
>2012-12-17 09:37:24,711 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:37:28,863 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:37:28,892 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4911
>2012-12-17 09:37:28,893 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4911, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 392dad84
>2012-12-17 09:37:28,915 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling started: taskId = ae0f46a6-da6f-4d41-b1a1-90ba3b0564ec
>2012-12-17 09:37:29,936 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended: taskId = ae0f46a6-da6f-4d41-b1a1-90ba3b0564ec task status = finished
>2012-12-17 09:37:29,954 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended, spm status: SPM
>2012-12-17 09:37:29,956 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ae0f46a6-da6f-4d41-b1a1-90ba3b0564ec), log id: 53831a7b
>2012-12-17 09:37:29,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMClearTaskVDSCommand, log id: 53831a7b
>2012-12-17 09:37:29,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@443fac12, log id: 392dad84
>2012-12-17 09:37:29,987 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:37:30,015 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [36643dd7] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:37:30,015 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6cf7efd2
>2012-12-17 09:37:30,040 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:37:30,149 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [2e298a4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:37:30,200 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 402563d3
>2012-12-17 09:37:30,201 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6d354812
>2012-12-17 09:37:30,221 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:30,404 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] FINISH, SpmStopVDSCommand, log id: 6d354812
>2012-12-17 09:37:30,458 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] FINISH, ResetIrsVDSCommand, log id: 402563d3
>2012-12-17 09:37:30,470 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 79bd14f4
>2012-12-17 09:37:30,470 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [2e298a4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:30,470 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 79bd14f4
>2012-12-17 09:37:31,420 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:37:31,471 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4912
>2012-12-17 09:37:31,473 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4912, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3b4ffea4
>2012-12-17 09:37:31,495 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] spmStart polling started: taskId = bbcc6eb4-d8ef-415f-8740-908f7a6ee040
>2012-12-17 09:37:32,517 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] spmStart polling ended: taskId = bbcc6eb4-d8ef-415f-8740-908f7a6ee040 task status = finished
>2012-12-17 09:37:32,535 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] spmStart polling ended, spm status: SPM
>2012-12-17 09:37:32,537 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bbcc6eb4-d8ef-415f-8740-908f7a6ee040), log id: 78da53c3
>2012-12-17 09:37:32,555 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, HSMClearTaskVDSCommand, log id: 78da53c3
>2012-12-17 09:37:32,555 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@45b16531, log id: 3b4ffea4
>2012-12-17 09:37:32,580 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:37:32,595 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:32,597 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5187094a
>2012-12-17 09:37:32,618 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5187094a
>2012-12-17 09:37:32,618 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6cf7efd2
>2012-12-17 09:37:32,618 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [36643dd7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:37:32,618 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 78730316
>2012-12-17 09:37:32,618 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:32,620 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1f5ec121
>2012-12-17 09:37:32,638 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1f5ec121
>2012-12-17 09:37:32,638 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 78730316
>2012-12-17 09:37:32,638 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [36643dd7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:37:34,911 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:37:41,514 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [5bc13aa6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:37:41,655 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:37:41,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3f24ac3a
>2012-12-17 09:37:41,698 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e7f58a1
>2012-12-17 09:37:41,718 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:42,018 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] FINISH, SpmStopVDSCommand, log id: 7e7f58a1
>2012-12-17 09:37:42,031 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] FINISH, ResetIrsVDSCommand, log id: 3f24ac3a
>2012-12-17 09:37:42,045 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 256e51a4
>2012-12-17 09:37:42,045 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:42,045 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 256e51a4
>2012-12-17 09:37:45,099 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [76c72cb4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:37:52,117 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:37:52,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4913
>2012-12-17 09:37:52,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4913, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 48deaec4
>2012-12-17 09:37:52,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] spmStart polling started: taskId = 48a82fd4-23b7-45a1-a135-cad7b3c426c6
>2012-12-17 09:37:53,205 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] spmStart polling ended: taskId = 48a82fd4-23b7-45a1-a135-cad7b3c426c6 task status = finished
>2012-12-17 09:37:53,223 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] spmStart polling ended, spm status: SPM
>2012-12-17 09:37:53,225 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=48a82fd4-23b7-45a1-a135-cad7b3c426c6), log id: 466cb3ea
>2012-12-17 09:37:53,243 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, HSMClearTaskVDSCommand, log id: 466cb3ea
>2012-12-17 09:37:53,243 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@a97ebc5, log id: 48deaec4
>2012-12-17 09:37:53,268 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:37:53,326 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [134ba83e] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:37:53,326 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 464a9adc
>2012-12-17 09:37:53,339 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:37:53,523 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [7fbbf72c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:37:53,653 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: b9d6763
>2012-12-17 09:37:53,654 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5fe02130
>2012-12-17 09:37:53,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:53,833 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] FINISH, SpmStopVDSCommand, log id: 5fe02130
>2012-12-17 09:37:53,845 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] FINISH, ResetIrsVDSCommand, log id: b9d6763
>2012-12-17 09:37:53,856 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1cf80d4d
>2012-12-17 09:37:53,856 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7fbbf72c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:53,856 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1cf80d4d
>2012-12-17 09:37:53,930 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [134ba83e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:37:53,947 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [134ba83e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4914
>2012-12-17 09:37:53,948 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4914, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 68f47bb2
>2012-12-17 09:37:53,971 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] spmStart polling started: taskId = 67a50722-4cd2-4f67-afc2-77b541477239
>2012-12-17 09:37:54,993 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] spmStart polling ended: taskId = 67a50722-4cd2-4f67-afc2-77b541477239 task status = finished
>2012-12-17 09:37:55,012 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] spmStart polling ended, spm status: SPM
>2012-12-17 09:37:55,013 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=67a50722-4cd2-4f67-afc2-77b541477239), log id: 72a0ffea
>2012-12-17 09:37:55,031 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, HSMClearTaskVDSCommand, log id: 72a0ffea
>2012-12-17 09:37:55,031 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@28019671, log id: 68f47bb2
>2012-12-17 09:37:55,045 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [134ba83e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:37:55,075 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:55,077 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4b1bb827
>2012-12-17 09:37:55,098 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b1bb827
>2012-12-17 09:37:55,098 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 464a9adc
>2012-12-17 09:37:55,098 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [134ba83e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:37:55,098 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 674c5ea9
>2012-12-17 09:37:55,098 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:37:55,099 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6da1ae1
>2012-12-17 09:37:55,118 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6da1ae1
>2012-12-17 09:37:55,118 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 674c5ea9
>2012-12-17 09:37:55,118 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [134ba83e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:37:55,350 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [3ad3ad88] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:38:03,979 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [22fa8e65] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:38:04,086 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [5f4cecb9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:38:04,145 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5668e9a
>2012-12-17 09:38:04,147 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3e59458b
>2012-12-17 09:38:04,166 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:38:04,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] FINISH, SpmStopVDSCommand, log id: 3e59458b
>2012-12-17 09:38:04,519 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] FINISH, ResetIrsVDSCommand, log id: 5668e9a
>2012-12-17 09:38:04,530 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 383cc203
>2012-12-17 09:38:04,530 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [5f4cecb9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:38:04,530 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 383cc203
>2012-12-17 09:38:06,788 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [78e614fb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:38:14,603 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3c4cd577] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:38:14,615 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3c4cd577] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4915
>2012-12-17 09:38:14,616 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4915, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 42877e07
>2012-12-17 09:38:14,638 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] spmStart polling started: taskId = 7198a7bf-f832-4c02-8498-22e320d0da29
>2012-12-17 09:38:15,660 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] spmStart polling ended: taskId = 7198a7bf-f832-4c02-8498-22e320d0da29 task status = finished
>2012-12-17 09:38:15,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] spmStart polling ended, spm status: SPM
>2012-12-17 09:38:15,679 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7198a7bf-f832-4c02-8498-22e320d0da29), log id: 5bda95f1
>2012-12-17 09:38:15,697 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] FINISH, HSMClearTaskVDSCommand, log id: 5bda95f1
>2012-12-17 09:38:15,697 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6baa3053, log id: 42877e07
>2012-12-17 09:38:15,713 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3c4cd577] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:38:16,159 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [786aa7bb] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:38:16,159 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5957bc7a
>2012-12-17 09:38:16,177 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3c4cd577] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:38:16,287 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [2d67b14a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:38:16,358 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6f4c6fb6
>2012-12-17 09:38:16,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 297af071
>2012-12-17 09:38:16,380 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:38:16,495 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SpmStopVDSCommand, log id: 297af071
>2012-12-17 09:38:16,514 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, ResetIrsVDSCommand, log id: 6f4c6fb6
>2012-12-17 09:38:16,525 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6f5d3871
>2012-12-17 09:38:16,525 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:38:16,525 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6f5d3871
>2012-12-17 09:38:16,598 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:38:16,606 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4916
>2012-12-17 09:38:16,607 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4916, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 72212314
>2012-12-17 09:38:16,629 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] spmStart polling started: taskId = e5c64db9-d242-4cde-8c96-d1d69a6317a0
>2012-12-17 09:38:17,650 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] spmStart polling ended: taskId = e5c64db9-d242-4cde-8c96-d1d69a6317a0 task status = finished
>2012-12-17 09:38:17,669 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] spmStart polling ended, spm status: SPM
>2012-12-17 09:38:17,670 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e5c64db9-d242-4cde-8c96-d1d69a6317a0), log id: 1dc46dfe
>2012-12-17 09:38:17,688 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, HSMClearTaskVDSCommand, log id: 1dc46dfe
>2012-12-17 09:38:17,688 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4b114df6, log id: 72212314
>2012-12-17 09:38:17,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:38:17,760 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:38:17,762 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d6bfdfb
>2012-12-17 09:38:17,781 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d6bfdfb
>2012-12-17 09:38:17,817 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5957bc7a
>2012-12-17 09:38:17,817 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [786aa7bb] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:38:17,817 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3432facc
>2012-12-17 09:38:17,817 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:38:17,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d9b99d8
>2012-12-17 09:38:17,841 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d9b99d8
>2012-12-17 09:38:17,841 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3432facc
>2012-12-17 09:38:17,841 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [786aa7bb] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:38:26,743 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [f37b622] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:38:26,873 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [130592b9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:38:26,918 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [130592b9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 79e9d8
>2012-12-17 09:38:26,919 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [130592b9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4b8b526e
>2012-12-17 09:38:26,939 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [130592b9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:38:27,103 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [130592b9] FINISH, SpmStopVDSCommand, log id: 4b8b526e
>2012-12-17 09:38:27,117 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [130592b9] FINISH, ResetIrsVDSCommand, log id: 79e9d8
>2012-12-17 09:38:27,130 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [130592b9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 77cd7e0c
>2012-12-17 09:38:27,130 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [130592b9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:38:27,130 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [130592b9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 77cd7e0c
>2012-12-17 09:38:27,300 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:38:37,191 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [40c8f45e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:38:37,202 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [40c8f45e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4917
>2012-12-17 09:38:37,204 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4917, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: bbe182d
>2012-12-17 09:38:37,226 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] spmStart polling started: taskId = b1820aba-b09c-491d-8f7b-52b30b119817
>2012-12-17 09:38:38,256 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] spmStart polling ended: taskId = b1820aba-b09c-491d-8f7b-52b30b119817 task status = finished
>2012-12-17 09:38:38,275 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] spmStart polling ended, spm status: SPM
>2012-12-17 09:38:38,276 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b1820aba-b09c-491d-8f7b-52b30b119817), log id: 2b1e2ff
>2012-12-17 09:38:38,295 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] FINISH, HSMClearTaskVDSCommand, log id: 2b1e2ff
>2012-12-17 09:38:38,295 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6cadd4ba, log id: bbe182d
>2012-12-17 09:38:38,326 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [40c8f45e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:38:38,386 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [5117cf2d] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:38:38,386 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 47e51e87
>2012-12-17 09:38:38,444 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [40c8f45e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:38:38,837 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [41718aec] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:38:39,094 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7c6d9b2b
>2012-12-17 09:38:39,096 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e047b59
>2012-12-17 09:38:39,115 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [41718aec] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:38:39,255 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, SpmStopVDSCommand, log id: 7e047b59
>2012-12-17 09:38:39,317 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, ResetIrsVDSCommand, log id: 7c6d9b2b
>2012-12-17 09:38:39,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: b67c433
>2012-12-17 09:38:39,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:38:39,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, MarkPoolInReconstructModeVDSCommand, log id: b67c433
>2012-12-17 09:38:39,412 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [5117cf2d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:38:39,425 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [5117cf2d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4918
>2012-12-17 09:38:39,453 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4918, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: eeeec7f
>2012-12-17 09:38:39,474 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] spmStart polling started: taskId = de9cfe28-2953-430a-a97e-21497e909a9e
>2012-12-17 09:38:40,496 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] spmStart polling ended: taskId = de9cfe28-2953-430a-a97e-21497e909a9e task status = finished
>2012-12-17 09:38:40,515 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] spmStart polling ended, spm status: SPM
>2012-12-17 09:38:40,517 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=de9cfe28-2953-430a-a97e-21497e909a9e), log id: 35401ef
>2012-12-17 09:38:40,535 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, HSMClearTaskVDSCommand, log id: 35401ef
>2012-12-17 09:38:40,535 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4af39e, log id: eeeec7f
>2012-12-17 09:38:40,773 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [5117cf2d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:38:40,831 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:38:40,835 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1e291d7f
>2012-12-17 09:38:40,855 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1e291d7f
>2012-12-17 09:38:40,855 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 47e51e87
>2012-12-17 09:38:40,855 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [5117cf2d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:38:40,855 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1d4f8c51
>2012-12-17 09:38:40,855 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:38:40,856 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4ff391cd
>2012-12-17 09:38:40,875 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4ff391cd
>2012-12-17 09:38:40,875 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1d4f8c51
>2012-12-17 09:38:40,875 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [5117cf2d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:38:47,843 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5c6bc06e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:38:49,459 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6728cccd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:38:49,597 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-56) [65ba9edd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:38:49,659 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [65ba9edd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 21b5abde
>2012-12-17 09:38:49,661 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [65ba9edd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 60dedd2a
>2012-12-17 09:38:49,681 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [65ba9edd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:38:49,844 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [65ba9edd] FINISH, SpmStopVDSCommand, log id: 60dedd2a
>2012-12-17 09:38:49,858 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [65ba9edd] FINISH, ResetIrsVDSCommand, log id: 21b5abde
>2012-12-17 09:38:49,879 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [65ba9edd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a7f833c
>2012-12-17 09:38:49,879 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [65ba9edd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:38:49,879 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [65ba9edd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a7f833c
>2012-12-17 09:38:58,063 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:38:59,967 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [134cc8af] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:38:59,976 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [134cc8af] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4919
>2012-12-17 09:38:59,977 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4919, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 474b2943
>2012-12-17 09:39:00,000 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] spmStart polling started: taskId = 923a75bc-7ba6-4a9a-85ec-80e45bc51eda
>2012-12-17 09:39:01,023 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] spmStart polling ended: taskId = 923a75bc-7ba6-4a9a-85ec-80e45bc51eda task status = finished
>2012-12-17 09:39:01,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] spmStart polling ended, spm status: SPM
>2012-12-17 09:39:01,043 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=923a75bc-7ba6-4a9a-85ec-80e45bc51eda), log id: 7441893f
>2012-12-17 09:39:01,062 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] FINISH, HSMClearTaskVDSCommand, log id: 7441893f
>2012-12-17 09:39:01,062 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7a99c2b9, log id: 474b2943
>2012-12-17 09:39:01,300 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [134cc8af] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:39:01,382 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [7790038] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:39:01,382 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3ee97ebf
>2012-12-17 09:39:01,400 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [134cc8af] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:39:01,506 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [16841b99] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:39:01,555 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3853af67
>2012-12-17 09:39:01,557 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 317e1109
>2012-12-17 09:39:01,577 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [16841b99] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:02,275 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, SpmStopVDSCommand, log id: 317e1109
>2012-12-17 09:39:02,297 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, ResetIrsVDSCommand, log id: 3853af67
>2012-12-17 09:39:02,308 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 13247aac
>2012-12-17 09:39:02,308 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:02,308 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 13247aac
>2012-12-17 09:39:02,361 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:39:02,377 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4920
>2012-12-17 09:39:02,378 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4920, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2b032fe6
>2012-12-17 09:39:02,400 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] spmStart polling started: taskId = 9faa6ca9-97d9-449e-839f-1965bcb4cbca
>2012-12-17 09:39:03,423 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] spmStart polling ended: taskId = 9faa6ca9-97d9-449e-839f-1965bcb4cbca task status = finished
>2012-12-17 09:39:03,441 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] spmStart polling ended, spm status: SPM
>2012-12-17 09:39:03,443 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9faa6ca9-97d9-449e-839f-1965bcb4cbca), log id: 3e3888c6
>2012-12-17 09:39:03,462 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, HSMClearTaskVDSCommand, log id: 3e3888c6
>2012-12-17 09:39:03,462 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@719b9f13, log id: 2b032fe6
>2012-12-17 09:39:03,478 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:39:03,500 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:03,502 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 44b929ee
>2012-12-17 09:39:03,520 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 44b929ee
>2012-12-17 09:39:03,521 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3ee97ebf
>2012-12-17 09:39:03,521 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [7790038] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:39:03,521 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 586feb81
>2012-12-17 09:39:03,521 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:03,522 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1a5f0eb4
>2012-12-17 09:39:03,541 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1a5f0eb4
>2012-12-17 09:39:03,541 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 586feb81
>2012-12-17 09:39:03,541 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [7790038] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:39:08,299 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [515f648d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:39:12,421 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [6157bb5e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:39:12,602 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [4e8c7167] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:39:12,702 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [4e8c7167] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 39b12985
>2012-12-17 09:39:12,704 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [4e8c7167] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 18389e7
>2012-12-17 09:39:12,723 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [4e8c7167] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:12,919 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [4e8c7167] FINISH, SpmStopVDSCommand, log id: 18389e7
>2012-12-17 09:39:12,936 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [4e8c7167] FINISH, ResetIrsVDSCommand, log id: 39b12985
>2012-12-17 09:39:12,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [4e8c7167] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a340a83
>2012-12-17 09:39:12,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [4e8c7167] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:12,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [4e8c7167] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a340a83
>2012-12-17 09:39:18,488 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:39:23,011 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [6c33f0df] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:39:23,025 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [6c33f0df] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4921
>2012-12-17 09:39:23,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4921, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1f958afd
>2012-12-17 09:39:23,052 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] spmStart polling started: taskId = 94eed677-a039-46da-9c4a-d838648dffde
>2012-12-17 09:39:24,075 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] spmStart polling ended: taskId = 94eed677-a039-46da-9c4a-d838648dffde task status = finished
>2012-12-17 09:39:24,094 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] spmStart polling ended, spm status: SPM
>2012-12-17 09:39:24,095 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=94eed677-a039-46da-9c4a-d838648dffde), log id: 41474783
>2012-12-17 09:39:24,114 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] FINISH, HSMClearTaskVDSCommand, log id: 41474783
>2012-12-17 09:39:24,114 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2ea80af0, log id: 1f958afd
>2012-12-17 09:39:24,124 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [6c33f0df] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:39:24,154 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [492c9849] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:39:24,154 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2380c43f
>2012-12-17 09:39:24,187 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [6c33f0df] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:39:24,318 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-51) [56fd87e7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:39:24,393 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5ced27b4
>2012-12-17 09:39:24,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3fa72742
>2012-12-17 09:39:24,414 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:24,551 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, SpmStopVDSCommand, log id: 3fa72742
>2012-12-17 09:39:24,578 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, ResetIrsVDSCommand, log id: 5ced27b4
>2012-12-17 09:39:24,589 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 15e2412
>2012-12-17 09:39:24,589 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:24,589 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 15e2412
>2012-12-17 09:39:24,689 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:39:24,718 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4922
>2012-12-17 09:39:24,719 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4922, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5023ea79
>2012-12-17 09:39:24,742 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] spmStart polling started: taskId = 00247663-9254-464d-85c5-794d10c6dd57
>2012-12-17 09:39:25,764 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] spmStart polling ended: taskId = 00247663-9254-464d-85c5-794d10c6dd57 task status = finished
>2012-12-17 09:39:25,783 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] spmStart polling ended, spm status: SPM
>2012-12-17 09:39:25,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=00247663-9254-464d-85c5-794d10c6dd57), log id: 17205f18
>2012-12-17 09:39:25,802 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, HSMClearTaskVDSCommand, log id: 17205f18
>2012-12-17 09:39:25,802 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4e432c84, log id: 5023ea79
>2012-12-17 09:39:26,325 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:39:26,445 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:26,447 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5403a555
>2012-12-17 09:39:26,468 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5403a555
>2012-12-17 09:39:26,468 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2380c43f
>2012-12-17 09:39:26,468 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [492c9849] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:39:26,468 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3b5b9a8a
>2012-12-17 09:39:26,468 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:26,470 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2565775e
>2012-12-17 09:39:26,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2565775e
>2012-12-17 09:39:26,489 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3b5b9a8a
>2012-12-17 09:39:26,489 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [492c9849] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:39:28,770 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:39:34,751 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [8d9c42c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:39:34,911 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [34a366b8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:39:35,006 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 228c02e9
>2012-12-17 09:39:35,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4ce32a47
>2012-12-17 09:39:35,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:35,186 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SpmStopVDSCommand, log id: 4ce32a47
>2012-12-17 09:39:35,209 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, ResetIrsVDSCommand, log id: 228c02e9
>2012-12-17 09:39:35,220 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5dbf1440
>2012-12-17 09:39:35,220 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:35,220 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5dbf1440
>2012-12-17 09:39:39,018 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [248935dc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:39:45,293 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:39:45,412 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4923
>2012-12-17 09:39:45,415 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4923, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5926b6d9
>2012-12-17 09:39:45,439 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling started: taskId = 6f469cfe-def8-45d4-a6da-f2f7e6a49b63
>2012-12-17 09:39:46,461 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling ended: taskId = 6f469cfe-def8-45d4-a6da-f2f7e6a49b63 task status = finished
>2012-12-17 09:39:46,479 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling ended, spm status: SPM
>2012-12-17 09:39:46,481 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6f469cfe-def8-45d4-a6da-f2f7e6a49b63), log id: 6d0d3e4c
>2012-12-17 09:39:46,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, HSMClearTaskVDSCommand, log id: 6d0d3e4c
>2012-12-17 09:39:46,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@752783, log id: 5926b6d9
>2012-12-17 09:39:47,539 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:39:47,864 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:39:47,865 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 50491293
>2012-12-17 09:39:47,977 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:39:49,727 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:39:49,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 226b3738
>2012-12-17 09:39:49,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 23a21faf
>2012-12-17 09:39:49,849 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:49,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] FINISH, SpmStopVDSCommand, log id: 23a21faf
>2012-12-17 09:39:50,008 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] FINISH, ResetIrsVDSCommand, log id: 226b3738
>2012-12-17 09:39:50,019 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 329ef07e
>2012-12-17 09:39:50,019 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:50,019 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 329ef07e
>2012-12-17 09:39:50,063 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:39:50,126 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4924
>2012-12-17 09:39:50,127 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4924, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 61d86699
>2012-12-17 09:39:50,151 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling started: taskId = b219dc71-6476-445d-a052-1fa3348e6cb8
>2012-12-17 09:39:51,172 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling ended: taskId = b219dc71-6476-445d-a052-1fa3348e6cb8 task status = finished
>2012-12-17 09:39:51,190 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling ended, spm status: SPM
>2012-12-17 09:39:51,192 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b219dc71-6476-445d-a052-1fa3348e6cb8), log id: 2802bea5
>2012-12-17 09:39:51,210 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMClearTaskVDSCommand, log id: 2802bea5
>2012-12-17 09:39:51,210 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4c325728, log id: 61d86699
>2012-12-17 09:39:51,712 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:39:52,185 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:52,190 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2356f486
>2012-12-17 09:39:52,209 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2356f486
>2012-12-17 09:39:52,209 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 50491293
>2012-12-17 09:39:52,209 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:39:52,209 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 21cf9b8a
>2012-12-17 09:39:52,209 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:39:52,211 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7948415b
>2012-12-17 09:39:52,229 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7948415b
>2012-12-17 09:39:52,229 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 21cf9b8a
>2012-12-17 09:39:52,229 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:40:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [24a780e1] Checking autorecoverable hosts
>2012-12-17 09:40:00,094 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [24a780e1] Autorecovering 0 hosts
>2012-12-17 09:40:00,094 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [24a780e1] Checking autorecoverable hosts done
>2012-12-17 09:40:00,094 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [24a780e1] Autorecovering storage domains is disabled, skipping
>2012-12-17 09:40:00,279 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [136706cf] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:40:00,359 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1ed39251] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:40:02,670 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:40:02,802 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 210c0d3e
>2012-12-17 09:40:02,804 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 33618c62
>2012-12-17 09:40:02,825 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:40:03,017 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] FINISH, SpmStopVDSCommand, log id: 33618c62
>2012-12-17 09:40:03,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] FINISH, ResetIrsVDSCommand, log id: 210c0d3e
>2012-12-17 09:40:03,058 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4de7108b
>2012-12-17 09:40:03,058 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:40:03,058 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4de7108b
>2012-12-17 09:40:11,389 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [46833f1f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:40:13,138 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [56a0d824] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:40:13,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [56a0d824] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4925
>2012-12-17 09:40:13,159 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4925, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2a31f6af
>2012-12-17 09:40:13,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] spmStart polling started: taskId = 694138e5-811e-499a-a432-6e619c50baee
>2012-12-17 09:40:14,204 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] spmStart polling ended: taskId = 694138e5-811e-499a-a432-6e619c50baee task status = finished
>2012-12-17 09:40:14,222 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] spmStart polling ended, spm status: SPM
>2012-12-17 09:40:14,224 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=694138e5-811e-499a-a432-6e619c50baee), log id: 5de11455
>2012-12-17 09:40:14,243 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] FINISH, HSMClearTaskVDSCommand, log id: 5de11455
>2012-12-17 09:40:14,243 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@63fe0899, log id: 2a31f6af
>2012-12-17 09:40:14,404 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [56a0d824] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:40:14,466 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [492c9849] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:40:14,466 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 40ff0f06
>2012-12-17 09:40:14,486 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [56a0d824] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:40:14,602 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [7c8941c5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:40:14,676 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7c5f9f23
>2012-12-17 09:40:14,677 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4245b538
>2012-12-17 09:40:14,697 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:40:14,839 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, SpmStopVDSCommand, log id: 4245b538
>2012-12-17 09:40:14,854 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, ResetIrsVDSCommand, log id: 7c5f9f23
>2012-12-17 09:40:14,864 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2fe70786
>2012-12-17 09:40:14,864 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7c8941c5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:40:14,864 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2fe70786
>2012-12-17 09:40:14,918 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:40:14,936 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4926
>2012-12-17 09:40:14,937 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4926, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 513a74f5
>2012-12-17 09:40:14,958 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] spmStart polling started: taskId = 9be41fa0-52a2-43e5-8498-9d0b202bdf70
>2012-12-17 09:40:15,980 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] spmStart polling ended: taskId = 9be41fa0-52a2-43e5-8498-9d0b202bdf70 task status = finished
>2012-12-17 09:40:15,998 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] spmStart polling ended, spm status: SPM
>2012-12-17 09:40:15,999 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9be41fa0-52a2-43e5-8498-9d0b202bdf70), log id: 175563b1
>2012-12-17 09:40:16,017 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, HSMClearTaskVDSCommand, log id: 175563b1
>2012-12-17 09:40:16,017 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6704056e, log id: 513a74f5
>2012-12-17 09:40:16,677 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:40:16,695 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:40:16,700 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 16ae9298
>2012-12-17 09:40:16,724 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 16ae9298
>2012-12-17 09:40:16,724 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 40ff0f06
>2012-12-17 09:40:16,724 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [492c9849] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:40:16,724 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5365997b
>2012-12-17 09:40:16,724 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:40:16,726 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4b3756f9
>2012-12-17 09:40:16,745 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b3756f9
>2012-12-17 09:40:16,745 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5365997b
>2012-12-17 09:40:16,745 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [492c9849] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:40:21,830 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:40:24,962 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:40:25,087 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [4549d797] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:40:25,148 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [4549d797] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 40398b69
>2012-12-17 09:40:25,150 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [4549d797] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7c325aff
>2012-12-17 09:40:25,170 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [4549d797] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:40:25,546 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [4549d797] FINISH, SpmStopVDSCommand, log id: 7c325aff
>2012-12-17 09:40:25,639 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [4549d797] FINISH, ResetIrsVDSCommand, log id: 40398b69
>2012-12-17 09:40:25,650 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [4549d797] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 30072f7e
>2012-12-17 09:40:25,650 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [4549d797] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:40:25,650 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [4549d797] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 30072f7e
>2012-12-17 09:40:32,117 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:40:36,304 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [280f54d2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:40:36,649 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [280f54d2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4927
>2012-12-17 09:40:36,696 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4927, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 135a16aa
>2012-12-17 09:40:36,720 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] spmStart polling started: taskId = 013c5443-5ac7-413d-803d-ed2984f7a0a0
>2012-12-17 09:40:37,743 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] spmStart polling ended: taskId = 013c5443-5ac7-413d-803d-ed2984f7a0a0 task status = finished
>2012-12-17 09:40:37,762 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] spmStart polling ended, spm status: SPM
>2012-12-17 09:40:37,763 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=013c5443-5ac7-413d-803d-ed2984f7a0a0), log id: 467c073a
>2012-12-17 09:40:37,781 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] FINISH, HSMClearTaskVDSCommand, log id: 467c073a
>2012-12-17 09:40:37,781 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78be94fc, log id: 135a16aa
>2012-12-17 09:40:37,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [280f54d2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:40:37,864 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [346a0820] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:40:37,864 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 31a53e04
>2012-12-17 09:40:37,870 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [280f54d2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:40:37,979 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [6999432f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:40:38,080 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: cbea4cf
>2012-12-17 09:40:38,082 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 50ed6a7d
>2012-12-17 09:40:38,102 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [6999432f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:40:38,244 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, SpmStopVDSCommand, log id: 50ed6a7d
>2012-12-17 09:40:38,254 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, ResetIrsVDSCommand, log id: cbea4cf
>2012-12-17 09:40:38,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 43e7decd
>2012-12-17 09:40:38,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [6999432f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:40:38,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 43e7decd
>2012-12-17 09:40:38,332 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:40:38,345 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4928
>2012-12-17 09:40:38,346 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4928, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 45c62a06
>2012-12-17 09:40:38,369 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling started: taskId = a71467b6-5896-435a-abee-69dcd4449a38
>2012-12-17 09:40:39,391 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling ended: taskId = a71467b6-5896-435a-abee-69dcd4449a38 task status = finished
>2012-12-17 09:40:39,410 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling ended, spm status: SPM
>2012-12-17 09:40:39,411 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a71467b6-5896-435a-abee-69dcd4449a38), log id: 5f8816d7
>2012-12-17 09:40:39,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, HSMClearTaskVDSCommand, log id: 5f8816d7
>2012-12-17 09:40:39,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5f93ae40, log id: 45c62a06
>2012-12-17 09:40:39,442 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:40:39,455 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:40:39,459 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 61eb85ae
>2012-12-17 09:40:39,478 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 61eb85ae
>2012-12-17 09:40:39,478 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 31a53e04
>2012-12-17 09:40:39,478 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [346a0820] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:40:39,478 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3757985e
>2012-12-17 09:40:39,478 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:40:39,479 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3a813577
>2012-12-17 09:40:39,497 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3a813577
>2012-12-17 09:40:39,497 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3757985e
>2012-12-17 09:40:39,497 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [346a0820] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:40:42,411 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:40:48,375 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [24a780e1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:40:48,558 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [511011e7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:40:48,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [511011e7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 42b3a8dd
>2012-12-17 09:40:48,619 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [511011e7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 49f0fe06
>2012-12-17 09:40:48,638 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [511011e7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:40:49,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [511011e7] FINISH, SpmStopVDSCommand, log id: 49f0fe06
>2012-12-17 09:40:49,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [511011e7] FINISH, ResetIrsVDSCommand, log id: 42b3a8dd
>2012-12-17 09:40:49,387 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [511011e7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 422b1202
>2012-12-17 09:40:49,387 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [511011e7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:40:49,387 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [511011e7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 422b1202
>2012-12-17 09:40:52,646 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [680acadd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:40:59,458 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [76c72cb4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:40:59,619 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [76c72cb4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4929
>2012-12-17 09:40:59,621 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4929, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2644b0d2
>2012-12-17 09:40:59,644 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] spmStart polling started: taskId = cdd22f2c-daaf-4980-8655-4cf7a396558a
>2012-12-17 09:41:00,665 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] spmStart polling ended: taskId = cdd22f2c-daaf-4980-8655-4cf7a396558a task status = finished
>2012-12-17 09:41:00,683 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] spmStart polling ended, spm status: SPM
>2012-12-17 09:41:00,684 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cdd22f2c-daaf-4980-8655-4cf7a396558a), log id: 5048f852
>2012-12-17 09:41:00,702 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] FINISH, HSMClearTaskVDSCommand, log id: 5048f852
>2012-12-17 09:41:00,702 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@66a6f8ab, log id: 2644b0d2
>2012-12-17 09:41:01,318 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [76c72cb4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:41:02,193 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2c554bbe] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:41:02,193 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 42e94f89
>2012-12-17 09:41:02,307 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [76c72cb4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:41:02,579 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [33bbf9d8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:41:02,655 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 48f4915e
>2012-12-17 09:41:02,658 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1f94a078
>2012-12-17 09:41:02,685 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:02,818 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, SpmStopVDSCommand, log id: 1f94a078
>2012-12-17 09:41:02,862 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, ResetIrsVDSCommand, log id: 48f4915e
>2012-12-17 09:41:02,874 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e9a3db7
>2012-12-17 09:41:02,874 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [33bbf9d8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:02,874 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e9a3db7
>2012-12-17 09:41:02,956 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [519a0bb9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:41:03,016 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:41:03,045 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4930
>2012-12-17 09:41:03,047 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4930, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b4a9679
>2012-12-17 09:41:03,069 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] spmStart polling started: taskId = f7293709-41ed-403b-8c89-f2bde3dcc0d3
>2012-12-17 09:41:04,091 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] spmStart polling ended: taskId = f7293709-41ed-403b-8c89-f2bde3dcc0d3 task status = finished
>2012-12-17 09:41:04,110 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] spmStart polling ended, spm status: SPM
>2012-12-17 09:41:04,111 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f7293709-41ed-403b-8c89-f2bde3dcc0d3), log id: 6cfb447d
>2012-12-17 09:41:04,130 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, HSMClearTaskVDSCommand, log id: 6cfb447d
>2012-12-17 09:41:04,130 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@644da270, log id: 6b4a9679
>2012-12-17 09:41:04,144 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:41:04,169 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:04,173 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 65130b82
>2012-12-17 09:41:04,192 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 65130b82
>2012-12-17 09:41:04,192 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 42e94f89
>2012-12-17 09:41:04,192 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2c554bbe] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:41:04,192 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4558b7a5
>2012-12-17 09:41:04,192 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:04,193 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 22bf19c4
>2012-12-17 09:41:04,194 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [5117cf2d] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:41:04,194 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5027b43b
>2012-12-17 09:41:04,212 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 22bf19c4
>2012-12-17 09:41:04,212 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4558b7a5
>2012-12-17 09:41:04,212 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2c554bbe] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:41:04,212 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:04,214 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2d6e0662
>2012-12-17 09:41:04,232 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2d6e0662
>2012-12-17 09:41:04,232 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5027b43b
>2012-12-17 09:41:04,232 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [5117cf2d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:41:04,232 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1265233b
>2012-12-17 09:41:04,232 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:04,233 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4a6cdffb
>2012-12-17 09:41:04,252 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4a6cdffb
>2012-12-17 09:41:04,252 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1265233b
>2012-12-17 09:41:04,252 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [5117cf2d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:41:13,033 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:41:13,143 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-80) [29a5a9b6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:41:13,319 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6871356b
>2012-12-17 09:41:13,321 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: b4f351a
>2012-12-17 09:41:13,342 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:13,498 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] FINISH, SpmStopVDSCommand, log id: b4f351a
>2012-12-17 09:41:13,512 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] FINISH, ResetIrsVDSCommand, log id: 6871356b
>2012-12-17 09:41:13,523 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1e5a90e1
>2012-12-17 09:41:13,523 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [29a5a9b6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:13,523 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1e5a90e1
>2012-12-17 09:41:23,448 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:41:23,571 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:41:23,651 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4931
>2012-12-17 09:41:23,652 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4931, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 53dc932
>2012-12-17 09:41:23,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] spmStart polling started: taskId = e8b15d7f-e384-4cd9-90e4-aaee968bba41
>2012-12-17 09:41:24,696 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] spmStart polling ended: taskId = e8b15d7f-e384-4cd9-90e4-aaee968bba41 task status = finished
>2012-12-17 09:41:24,715 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] spmStart polling ended, spm status: SPM
>2012-12-17 09:41:24,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e8b15d7f-e384-4cd9-90e4-aaee968bba41), log id: 60d54f2c
>2012-12-17 09:41:24,735 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] FINISH, HSMClearTaskVDSCommand, log id: 60d54f2c
>2012-12-17 09:41:24,735 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5441035e, log id: 53dc932
>2012-12-17 09:41:24,748 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:41:24,800 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) [16841b99] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:41:24,801 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 593bd992
>2012-12-17 09:41:24,821 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:41:24,947 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [21d8114] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:41:25,043 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [21d8114] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 12f7da04
>2012-12-17 09:41:25,044 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [21d8114] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7188f6da
>2012-12-17 09:41:25,064 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [21d8114] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:25,261 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [21d8114] FINISH, SpmStopVDSCommand, log id: 7188f6da
>2012-12-17 09:41:25,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [21d8114] FINISH, ResetIrsVDSCommand, log id: 12f7da04
>2012-12-17 09:41:25,300 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [21d8114] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 9098fb
>2012-12-17 09:41:25,300 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [21d8114] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:25,300 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [21d8114] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 9098fb
>2012-12-17 09:41:25,341 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:41:25,362 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4932
>2012-12-17 09:41:25,364 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4932, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 457f935b
>2012-12-17 09:41:25,388 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [16841b99] spmStart polling started: taskId = 8cd371fd-6acd-4ce3-bce9-916bdf7e7da3
>2012-12-17 09:41:26,411 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [16841b99] spmStart polling ended: taskId = 8cd371fd-6acd-4ce3-bce9-916bdf7e7da3 task status = finished
>2012-12-17 09:41:26,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [16841b99] spmStart polling ended, spm status: SPM
>2012-12-17 09:41:26,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8cd371fd-6acd-4ce3-bce9-916bdf7e7da3), log id: 2cf50cfe
>2012-12-17 09:41:26,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, HSMClearTaskVDSCommand, log id: 2cf50cfe
>2012-12-17 09:41:26,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@c280ec1, log id: 457f935b
>2012-12-17 09:41:26,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:41:26,477 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:26,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 186ae989
>2012-12-17 09:41:26,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 186ae989
>2012-12-17 09:41:26,500 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 593bd992
>2012-12-17 09:41:26,500 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) [16841b99] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:41:26,500 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 130a2de5
>2012-12-17 09:41:26,500 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:26,501 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2e921bf7
>2012-12-17 09:41:26,520 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2e921bf7
>2012-12-17 09:41:26,520 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 130a2de5
>2012-12-17 09:41:26,520 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) [16841b99] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:41:33,633 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:41:35,409 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7db92791] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:41:36,169 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-45) [21122b75] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:41:36,369 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 107b686e
>2012-12-17 09:41:36,370 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 758722bc
>2012-12-17 09:41:36,390 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [21122b75] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:37,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, SpmStopVDSCommand, log id: 758722bc
>2012-12-17 09:41:37,257 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, ResetIrsVDSCommand, log id: 107b686e
>2012-12-17 09:41:37,269 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6215b8c8
>2012-12-17 09:41:37,269 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:37,269 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6215b8c8
>2012-12-17 09:41:43,849 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:41:47,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [588a5ec3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:41:47,403 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [588a5ec3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4933
>2012-12-17 09:41:47,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4933, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6ed4044
>2012-12-17 09:41:47,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] spmStart polling started: taskId = 968a8e2c-8e23-4ce1-b7c7-4014d9592311
>2012-12-17 09:41:49,472 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] spmStart polling ended: taskId = 968a8e2c-8e23-4ce1-b7c7-4014d9592311 task status = finished
>2012-12-17 09:41:49,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] spmStart polling ended, spm status: SPM
>2012-12-17 09:41:49,491 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=968a8e2c-8e23-4ce1-b7c7-4014d9592311), log id: 35d804fb
>2012-12-17 09:41:49,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] FINISH, HSMClearTaskVDSCommand, log id: 35d804fb
>2012-12-17 09:41:49,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@792f165a, log id: 6ed4044
>2012-12-17 09:41:49,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [588a5ec3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:41:49,565 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [41718aec] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:41:49,565 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4c3e7702
>2012-12-17 09:41:49,598 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [588a5ec3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:41:49,762 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [2420ee59] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:41:49,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [2420ee59] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 31e2f4ec
>2012-12-17 09:41:49,832 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [2420ee59] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5654a1db
>2012-12-17 09:41:49,852 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [2420ee59] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:49,985 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [2420ee59] FINISH, SpmStopVDSCommand, log id: 5654a1db
>2012-12-17 09:41:49,996 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [2420ee59] FINISH, ResetIrsVDSCommand, log id: 31e2f4ec
>2012-12-17 09:41:50,006 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [2420ee59] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 66d687a7
>2012-12-17 09:41:50,007 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [2420ee59] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:50,007 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [2420ee59] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 66d687a7
>2012-12-17 09:41:50,051 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:41:50,108 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4934
>2012-12-17 09:41:50,109 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4934, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 23733f42
>2012-12-17 09:41:50,131 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] spmStart polling started: taskId = def68268-145d-4162-ac04-a39494ddcf1f
>2012-12-17 09:41:51,154 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] spmStart polling ended: taskId = def68268-145d-4162-ac04-a39494ddcf1f task status = finished
>2012-12-17 09:41:51,172 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] spmStart polling ended, spm status: SPM
>2012-12-17 09:41:51,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=def68268-145d-4162-ac04-a39494ddcf1f), log id: 69e87359
>2012-12-17 09:41:51,192 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, HSMClearTaskVDSCommand, log id: 69e87359
>2012-12-17 09:41:51,192 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1bed0e61, log id: 23733f42
>2012-12-17 09:41:51,439 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:41:51,607 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:51,609 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3aa1bb7e
>2012-12-17 09:41:51,629 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3aa1bb7e
>2012-12-17 09:41:51,629 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4c3e7702
>2012-12-17 09:41:51,629 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [41718aec] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:41:51,630 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 501c6b44
>2012-12-17 09:41:51,630 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:41:51,631 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4eae36d9
>2012-12-17 09:41:51,654 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4eae36d9
>2012-12-17 09:41:51,654 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 501c6b44
>2012-12-17 09:41:51,654 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [41718aec] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:41:54,076 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:42:00,138 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [3623f38c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:42:00,349 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [44cd9139] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:42:00,466 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 18f8c8a0
>2012-12-17 09:42:00,467 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 166cbf3b
>2012-12-17 09:42:00,487 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:00,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, SpmStopVDSCommand, log id: 166cbf3b
>2012-12-17 09:42:01,023 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, ResetIrsVDSCommand, log id: 18f8c8a0
>2012-12-17 09:42:01,035 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 561a060c
>2012-12-17 09:42:01,035 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:01,035 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 561a060c
>2012-12-17 09:42:04,315 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:42:11,687 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:42:11,700 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4935
>2012-12-17 09:42:11,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4935, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e7a3a42
>2012-12-17 09:42:11,725 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] spmStart polling started: taskId = 9a6bcbbe-2719-4e69-943e-8cc942e23731
>2012-12-17 09:42:12,747 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] spmStart polling ended: taskId = 9a6bcbbe-2719-4e69-943e-8cc942e23731 task status = finished
>2012-12-17 09:42:12,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] spmStart polling ended, spm status: SPM
>2012-12-17 09:42:12,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9a6bcbbe-2719-4e69-943e-8cc942e23731), log id: 7428c28d
>2012-12-17 09:42:12,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, HSMClearTaskVDSCommand, log id: 7428c28d
>2012-12-17 09:42:12,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1ffcf28f, log id: 3e7a3a42
>2012-12-17 09:42:12,843 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:42:12,970 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [6999432f] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:42:12,970 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4723ad5e
>2012-12-17 09:42:13,015 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:42:13,383 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-36) [58bd42f7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:42:13,502 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 47b65888
>2012-12-17 09:42:13,503 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 50bcb4bc
>2012-12-17 09:42:13,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:13,706 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] FINISH, SpmStopVDSCommand, log id: 50bcb4bc
>2012-12-17 09:42:13,737 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] FINISH, ResetIrsVDSCommand, log id: 47b65888
>2012-12-17 09:42:13,749 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40d609cd
>2012-12-17 09:42:13,749 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [58bd42f7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:13,749 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40d609cd
>2012-12-17 09:42:13,847 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [6999432f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:42:13,860 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [6999432f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4936
>2012-12-17 09:42:13,862 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4936, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 19fb0bdd
>2012-12-17 09:42:13,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [6999432f] spmStart polling started: taskId = 6978f336-1b1b-42f4-b0e0-13b95f38c1f8
>2012-12-17 09:42:14,905 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [6999432f] spmStart polling ended: taskId = 6978f336-1b1b-42f4-b0e0-13b95f38c1f8 task status = finished
>2012-12-17 09:42:14,923 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [6999432f] spmStart polling ended, spm status: SPM
>2012-12-17 09:42:14,925 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6978f336-1b1b-42f4-b0e0-13b95f38c1f8), log id: 73ed37b7
>2012-12-17 09:42:14,943 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, HSMClearTaskVDSCommand, log id: 73ed37b7
>2012-12-17 09:42:14,943 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5ae50ad6, log id: 19fb0bdd
>2012-12-17 09:42:14,967 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [6999432f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:42:14,982 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:14,983 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3a249275
>2012-12-17 09:42:15,005 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3a249275
>2012-12-17 09:42:15,005 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4723ad5e
>2012-12-17 09:42:15,005 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [6999432f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:42:15,005 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 370b5d25
>2012-12-17 09:42:15,005 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:15,006 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 751db5e5
>2012-12-17 09:42:15,024 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 751db5e5
>2012-12-17 09:42:15,024 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 370b5d25
>2012-12-17 09:42:15,025 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [6999432f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:42:23,886 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [136706cf] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:42:24,062 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-95) [17eab1eb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:42:24,125 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7d8d0dfa
>2012-12-17 09:42:24,126 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7b736393
>2012-12-17 09:42:24,146 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:24,711 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [4e8c7167] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:42:24,783 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, SpmStopVDSCommand, log id: 7b736393
>2012-12-17 09:42:24,795 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, ResetIrsVDSCommand, log id: 7d8d0dfa
>2012-12-17 09:42:24,806 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3afbe695
>2012-12-17 09:42:24,806 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:24,806 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3afbe695
>2012-12-17 09:42:34,879 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [46833f1f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:42:34,903 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [46833f1f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4937
>2012-12-17 09:42:34,905 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4937, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 8d149eb
>2012-12-17 09:42:34,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] spmStart polling started: taskId = 01403bdd-352d-433d-a458-0ee2bd5d5814
>2012-12-17 09:42:35,955 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] spmStart polling ended: taskId = 01403bdd-352d-433d-a458-0ee2bd5d5814 task status = finished
>2012-12-17 09:42:35,973 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] spmStart polling ended, spm status: SPM
>2012-12-17 09:42:35,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=01403bdd-352d-433d-a458-0ee2bd5d5814), log id: 203d1c41
>2012-12-17 09:42:35,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] FINISH, HSMClearTaskVDSCommand, log id: 203d1c41
>2012-12-17 09:42:35,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@c09fff0, log id: 8d149eb
>2012-12-17 09:42:36,249 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [46833f1f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:42:36,368 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:42:36,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7f018369
>2012-12-17 09:42:36,385 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [46833f1f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:42:36,516 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:42:36,574 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 36bbeb2b
>2012-12-17 09:42:36,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: c106166
>2012-12-17 09:42:36,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:37,259 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, SpmStopVDSCommand, log id: c106166
>2012-12-17 09:42:37,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, ResetIrsVDSCommand, log id: 36bbeb2b
>2012-12-17 09:42:37,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2d34ec4
>2012-12-17 09:42:37,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:37,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2d34ec4
>2012-12-17 09:42:37,326 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:42:37,337 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4938
>2012-12-17 09:42:37,338 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4938, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 546cf249
>2012-12-17 09:42:37,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling started: taskId = 68db3bbe-2f67-4ac5-baca-aef8c39da9bb
>2012-12-17 09:42:38,385 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended: taskId = 68db3bbe-2f67-4ac5-baca-aef8c39da9bb task status = finished
>2012-12-17 09:42:38,403 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended, spm status: SPM
>2012-12-17 09:42:38,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=68db3bbe-2f67-4ac5-baca-aef8c39da9bb), log id: 6c3a00d9
>2012-12-17 09:42:38,423 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMClearTaskVDSCommand, log id: 6c3a00d9
>2012-12-17 09:42:38,423 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@540dde11, log id: 546cf249
>2012-12-17 09:42:38,455 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:42:38,471 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:38,559 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7de43fbb
>2012-12-17 09:42:38,578 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7de43fbb
>2012-12-17 09:42:38,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7f018369
>2012-12-17 09:42:38,579 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:42:38,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3c99f627
>2012-12-17 09:42:38,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:38,580 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: a14711d
>2012-12-17 09:42:38,599 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: a14711d
>2012-12-17 09:42:38,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3c99f627
>2012-12-17 09:42:38,599 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:42:45,142 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [3ad3ad88] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:42:47,366 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5c6bc06e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:42:47,469 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [6412070c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:42:47,512 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [6412070c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 23707a2a
>2012-12-17 09:42:47,514 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [6412070c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 74a5b561
>2012-12-17 09:42:47,534 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [6412070c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:47,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [6412070c] FINISH, SpmStopVDSCommand, log id: 74a5b561
>2012-12-17 09:42:47,679 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [6412070c] FINISH, ResetIrsVDSCommand, log id: 23707a2a
>2012-12-17 09:42:47,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [6412070c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 34dc3f45
>2012-12-17 09:42:47,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [6412070c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:47,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [6412070c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 34dc3f45
>2012-12-17 09:42:56,789 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [78e614fb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:42:57,735 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [9b1d86] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:42:57,799 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [9b1d86] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4939
>2012-12-17 09:42:57,801 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4939, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5530bba8
>2012-12-17 09:42:57,824 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] spmStart polling started: taskId = 48b429bd-8996-4b0d-adbf-9699f7a1b682
>2012-12-17 09:42:58,853 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] spmStart polling ended: taskId = 48b429bd-8996-4b0d-adbf-9699f7a1b682 task status = finished
>2012-12-17 09:42:58,875 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] spmStart polling ended, spm status: SPM
>2012-12-17 09:42:58,876 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=48b429bd-8996-4b0d-adbf-9699f7a1b682), log id: 2079d445
>2012-12-17 09:42:58,895 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] FINISH, HSMClearTaskVDSCommand, log id: 2079d445
>2012-12-17 09:42:58,895 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1e5f0eab, log id: 5530bba8
>2012-12-17 09:42:58,908 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [9b1d86] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:42:58,995 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [1ab078c7] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:42:58,995 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6518bb61
>2012-12-17 09:42:59,011 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [9b1d86] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:42:59,269 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-37) [24cb4742] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:42:59,317 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 75373532
>2012-12-17 09:42:59,319 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 67447da2
>2012-12-17 09:42:59,338 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:59,581 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, SpmStopVDSCommand, log id: 67447da2
>2012-12-17 09:42:59,615 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, ResetIrsVDSCommand, log id: 75373532
>2012-12-17 09:42:59,626 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 761ab201
>2012-12-17 09:42:59,626 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [24cb4742] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:42:59,626 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 761ab201
>2012-12-17 09:42:59,679 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:42:59,689 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4940
>2012-12-17 09:42:59,690 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4940, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6d76600c
>2012-12-17 09:42:59,713 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling started: taskId = 6e89644f-6aed-45f4-a4e4-5e453b9b0b3a
>2012-12-17 09:43:00,751 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling ended: taskId = 6e89644f-6aed-45f4-a4e4-5e453b9b0b3a task status = finished
>2012-12-17 09:43:00,782 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling ended, spm status: SPM
>2012-12-17 09:43:00,783 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6e89644f-6aed-45f4-a4e4-5e453b9b0b3a), log id: 7a05e4a0
>2012-12-17 09:43:00,806 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, HSMClearTaskVDSCommand, log id: 7a05e4a0
>2012-12-17 09:43:00,806 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@18e502f6, log id: 6d76600c
>2012-12-17 09:43:03,754 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:43:03,877 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:03,880 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 22303e79
>2012-12-17 09:43:03,901 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 22303e79
>2012-12-17 09:43:03,901 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6518bb61
>2012-12-17 09:43:03,901 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [1ab078c7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:43:03,902 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: d558b78
>2012-12-17 09:43:03,902 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:03,903 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2b54930f
>2012-12-17 09:43:03,921 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2b54930f
>2012-12-17 09:43:03,921 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: d558b78
>2012-12-17 09:43:03,921 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [1ab078c7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:43:07,016 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:43:09,717 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [2bb64a93] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:43:09,808 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [c06c0c1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:43:09,858 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 222ee3b3
>2012-12-17 09:43:09,859 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 14366847
>2012-12-17 09:43:09,879 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:10,023 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SpmStopVDSCommand, log id: 14366847
>2012-12-17 09:43:10,035 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, ResetIrsVDSCommand, log id: 222ee3b3
>2012-12-17 09:43:10,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3f4b5afa
>2012-12-17 09:43:10,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:10,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3f4b5afa
>2012-12-17 09:43:17,210 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [680acadd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:43:20,094 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5d1b6c83] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:43:20,108 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5d1b6c83] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4941
>2012-12-17 09:43:20,110 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4941, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62a19d2c
>2012-12-17 09:43:20,134 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] spmStart polling started: taskId = ccfdcad7-a4dc-4753-801c-d320e9527ee1
>2012-12-17 09:43:21,155 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] spmStart polling ended: taskId = ccfdcad7-a4dc-4753-801c-d320e9527ee1 task status = finished
>2012-12-17 09:43:21,173 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] spmStart polling ended, spm status: SPM
>2012-12-17 09:43:21,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ccfdcad7-a4dc-4753-801c-d320e9527ee1), log id: 647c3569
>2012-12-17 09:43:21,193 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] FINISH, HSMClearTaskVDSCommand, log id: 647c3569
>2012-12-17 09:43:21,193 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1696b409, log id: 62a19d2c
>2012-12-17 09:43:21,233 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5d1b6c83] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:43:21,297 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [127bfc] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:43:21,297 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3053d947
>2012-12-17 09:43:21,339 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5d1b6c83] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:43:21,447 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [1eb0b032] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:43:21,526 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [1eb0b032] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7d1c6315
>2012-12-17 09:43:21,527 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [1eb0b032] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 24703341
>2012-12-17 09:43:21,548 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [1eb0b032] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:21,688 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [1eb0b032] FINISH, SpmStopVDSCommand, log id: 24703341
>2012-12-17 09:43:21,702 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [1eb0b032] FINISH, ResetIrsVDSCommand, log id: 7d1c6315
>2012-12-17 09:43:21,713 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [1eb0b032] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 46809868
>2012-12-17 09:43:21,713 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1eb0b032] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:21,713 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [1eb0b032] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 46809868
>2012-12-17 09:43:22,261 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:43:22,775 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4942
>2012-12-17 09:43:22,777 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4942, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e1e2f3d
>2012-12-17 09:43:22,800 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] spmStart polling started: taskId = 0723a8af-cd03-4793-acc7-1b2eb4001990
>2012-12-17 09:43:23,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] spmStart polling ended: taskId = 0723a8af-cd03-4793-acc7-1b2eb4001990 task status = finished
>2012-12-17 09:43:23,843 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] spmStart polling ended, spm status: SPM
>2012-12-17 09:43:23,844 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0723a8af-cd03-4793-acc7-1b2eb4001990), log id: 760850b1
>2012-12-17 09:43:23,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, HSMClearTaskVDSCommand, log id: 760850b1
>2012-12-17 09:43:23,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1be9d101, log id: 3e1e2f3d
>2012-12-17 09:43:23,889 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:43:23,906 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:23,911 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 55e4e851
>2012-12-17 09:43:23,931 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 55e4e851
>2012-12-17 09:43:23,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3053d947
>2012-12-17 09:43:23,931 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [127bfc] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:43:23,932 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 8236d8d
>2012-12-17 09:43:23,932 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:23,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 60dd7b1
>2012-12-17 09:43:23,952 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 60dd7b1
>2012-12-17 09:43:23,952 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 8236d8d
>2012-12-17 09:43:23,952 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [127bfc] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:43:34,459 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:43:34,538 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:43:34,863 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [205b7df7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:43:35,083 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 49b76ca2
>2012-12-17 09:43:35,085 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1dbee4ec
>2012-12-17 09:43:35,105 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:35,264 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] FINISH, SpmStopVDSCommand, log id: 1dbee4ec
>2012-12-17 09:43:35,275 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] FINISH, ResetIrsVDSCommand, log id: 49b76ca2
>2012-12-17 09:43:35,286 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 48471be5
>2012-12-17 09:43:35,286 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [205b7df7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:35,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 48471be5
>2012-12-17 09:43:45,113 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [65ba9edd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:43:45,447 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [248935dc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:43:46,741 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [248935dc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4943
>2012-12-17 09:43:46,743 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [248935dc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4943, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 22480666
>2012-12-17 09:43:46,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [248935dc] spmStart polling started: taskId = d3f91098-761e-4218-b7fc-379c60df049a
>2012-12-17 09:43:47,788 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [248935dc] spmStart polling ended: taskId = d3f91098-761e-4218-b7fc-379c60df049a task status = finished
>2012-12-17 09:43:47,806 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [248935dc] spmStart polling ended, spm status: SPM
>2012-12-17 09:43:47,807 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [248935dc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d3f91098-761e-4218-b7fc-379c60df049a), log id: 1d26c9e0
>2012-12-17 09:43:47,826 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [248935dc] FINISH, HSMClearTaskVDSCommand, log id: 1d26c9e0
>2012-12-17 09:43:47,826 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [248935dc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d8fa013, log id: 22480666
>2012-12-17 09:43:47,839 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [248935dc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:43:47,877 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [6eb2e264] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:43:47,877 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1eb2e2e7
>2012-12-17 09:43:47,891 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [248935dc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:43:48,002 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [4d6b231f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:43:48,057 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [4d6b231f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 706f8ae9
>2012-12-17 09:43:48,058 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [4d6b231f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 29835bd3
>2012-12-17 09:43:48,078 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [4d6b231f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:48,344 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [4d6b231f] FINISH, SpmStopVDSCommand, log id: 29835bd3
>2012-12-17 09:43:48,355 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [4d6b231f] FINISH, ResetIrsVDSCommand, log id: 706f8ae9
>2012-12-17 09:43:48,366 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [4d6b231f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7bcd97cf
>2012-12-17 09:43:48,366 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [4d6b231f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:48,366 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [4d6b231f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7bcd97cf
>2012-12-17 09:43:48,420 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:43:48,434 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4944
>2012-12-17 09:43:48,435 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4944, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 475c3b06
>2012-12-17 09:43:48,458 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] spmStart polling started: taskId = db62bd5f-9100-4fe9-93c5-0aa612fdf22f
>2012-12-17 09:43:49,478 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] spmStart polling ended: taskId = db62bd5f-9100-4fe9-93c5-0aa612fdf22f task status = finished
>2012-12-17 09:43:49,497 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] spmStart polling ended, spm status: SPM
>2012-12-17 09:43:49,498 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=db62bd5f-9100-4fe9-93c5-0aa612fdf22f), log id: 5b204280
>2012-12-17 09:43:49,517 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, HSMClearTaskVDSCommand, log id: 5b204280
>2012-12-17 09:43:49,517 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@22ee103d, log id: 475c3b06
>2012-12-17 09:43:49,564 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:43:49,639 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:49,644 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6fb5f1e7
>2012-12-17 09:43:49,662 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6fb5f1e7
>2012-12-17 09:43:49,663 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1eb2e2e7
>2012-12-17 09:43:49,663 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [6eb2e264] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:43:49,663 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 121cadfd
>2012-12-17 09:43:49,663 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:49,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 56769e04
>2012-12-17 09:43:49,683 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 56769e04
>2012-12-17 09:43:49,683 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 121cadfd
>2012-12-17 09:43:49,683 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [6eb2e264] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:43:55,483 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [be98275] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:43:58,461 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [6999432f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:43:58,562 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [75733504] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:43:58,665 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1df005d8
>2012-12-17 09:43:58,667 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 121c9c73
>2012-12-17 09:43:58,688 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [75733504] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:58,878 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, SpmStopVDSCommand, log id: 121c9c73
>2012-12-17 09:43:58,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, ResetIrsVDSCommand, log id: 1df005d8
>2012-12-17 09:43:58,920 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 691e46ec
>2012-12-17 09:43:58,921 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:43:58,921 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 691e46ec
>2012-12-17 09:44:07,339 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:44:08,995 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [49192055] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:44:09,102 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [49192055] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4945
>2012-12-17 09:44:09,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49192055] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4945, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e581f7a
>2012-12-17 09:44:09,128 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49192055] spmStart polling started: taskId = 157b816a-85b7-4501-9bc5-a3bbe928f473
>2012-12-17 09:44:10,150 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49192055] spmStart polling ended: taskId = 157b816a-85b7-4501-9bc5-a3bbe928f473 task status = finished
>2012-12-17 09:44:10,176 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49192055] spmStart polling ended, spm status: SPM
>2012-12-17 09:44:10,177 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [49192055] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=157b816a-85b7-4501-9bc5-a3bbe928f473), log id: 44d47477
>2012-12-17 09:44:10,196 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [49192055] FINISH, HSMClearTaskVDSCommand, log id: 44d47477
>2012-12-17 09:44:10,196 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49192055] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@582a53ee, log id: 5e581f7a
>2012-12-17 09:44:10,214 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [49192055] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:44:10,287 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [26d9e89a] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:44:10,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6146335d
>2012-12-17 09:44:10,300 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [49192055] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:44:10,542 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [61938c83] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:44:13,083 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [61938c83] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 54857515
>2012-12-17 09:44:13,085 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [61938c83] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4af712bd
>2012-12-17 09:44:13,105 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [61938c83] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:44:13,318 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [61938c83] FINISH, SpmStopVDSCommand, log id: 4af712bd
>2012-12-17 09:44:18,817 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [61938c83] FINISH, ResetIrsVDSCommand, log id: 54857515
>2012-12-17 09:44:18,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [61938c83] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e083abc
>2012-12-17 09:44:18,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [61938c83] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:44:18,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [61938c83] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e083abc
>2012-12-17 09:44:19,524 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:44:19,542 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:44:19,612 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4946
>2012-12-17 09:44:19,614 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4946, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3f0a0e88
>2012-12-17 09:44:19,636 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling started: taskId = b296419b-b531-4b5c-91b7-93fa2c518e90
>2012-12-17 09:44:20,660 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling ended: taskId = b296419b-b531-4b5c-91b7-93fa2c518e90 task status = finished
>2012-12-17 09:44:20,679 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling ended, spm status: SPM
>2012-12-17 09:44:20,681 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b296419b-b531-4b5c-91b7-93fa2c518e90), log id: 30e4a457
>2012-12-17 09:44:20,699 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, HSMClearTaskVDSCommand, log id: 30e4a457
>2012-12-17 09:44:20,699 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@72e164d4, log id: 3f0a0e88
>2012-12-17 09:44:21,038 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:44:21,084 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:44:21,091 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7d6c592f
>2012-12-17 09:44:21,110 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7d6c592f
>2012-12-17 09:44:21,110 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6146335d
>2012-12-17 09:44:21,110 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [26d9e89a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:44:21,110 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 54535cfd
>2012-12-17 09:44:21,110 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:44:21,111 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4ab4d809
>2012-12-17 09:44:21,130 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4ab4d809
>2012-12-17 09:44:21,130 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 54535cfd
>2012-12-17 09:44:21,130 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [26d9e89a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:44:21,157 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-50) [33bbf9d8] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:44:21,157 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7e8b0e39
>2012-12-17 09:44:21,157 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:44:21,159 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 73cad873
>2012-12-17 09:44:21,177 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 73cad873
>2012-12-17 09:44:21,178 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7e8b0e39
>2012-12-17 09:44:21,178 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-50) [33bbf9d8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:44:21,178 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: a48a169
>2012-12-17 09:44:21,178 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:44:21,179 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 28e7f452
>2012-12-17 09:44:21,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 28e7f452
>2012-12-17 09:44:21,197 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a48a169
>2012-12-17 09:44:21,197 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-50) [33bbf9d8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:44:29,596 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [134ba83e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:44:30,252 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [79357c7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:44:30,575 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [79357c7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: ca20dda
>2012-12-17 09:44:30,577 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [79357c7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: c469688
>2012-12-17 09:44:30,597 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [79357c7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:44:30,724 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [79357c7] FINISH, SpmStopVDSCommand, log id: c469688
>2012-12-17 09:44:31,685 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [79357c7] FINISH, ResetIrsVDSCommand, log id: ca20dda
>2012-12-17 09:44:31,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [79357c7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: cf3c6ed
>2012-12-17 09:44:31,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [79357c7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:44:31,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [79357c7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: cf3c6ed
>2012-12-17 09:44:41,920 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:44:42,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4947
>2012-12-17 09:44:42,926 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4947, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3fb8788f
>2012-12-17 09:44:42,954 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] spmStart polling started: taskId = 4d8ed956-7613-4632-b933-a0c01d114758
>2012-12-17 09:44:43,977 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] spmStart polling ended: taskId = 4d8ed956-7613-4632-b933-a0c01d114758 task status = finished
>2012-12-17 09:44:43,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] spmStart polling ended, spm status: SPM
>2012-12-17 09:44:43,997 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4d8ed956-7613-4632-b933-a0c01d114758), log id: 35302b57
>2012-12-17 09:44:44,016 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, HSMClearTaskVDSCommand, log id: 35302b57
>2012-12-17 09:44:44,016 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6d6d7808, log id: 3fb8788f
>2012-12-17 09:44:45,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:44:48,466 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:44:48,767 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-4) [5fb78147] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:44:48,865 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1a05d906
>2012-12-17 09:44:48,866 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 133754ea
>2012-12-17 09:44:48,887 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:44:49,028 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SpmStopVDSCommand, log id: 133754ea
>2012-12-17 09:44:49,041 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, ResetIrsVDSCommand, log id: 1a05d906
>2012-12-17 09:44:49,051 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24ef34aa
>2012-12-17 09:44:49,051 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:44:49,051 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24ef34aa
>2012-12-17 09:44:53,366 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:44:59,097 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6f4b2c75] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:44:59,121 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6f4b2c75] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4948
>2012-12-17 09:44:59,122 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4948, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5edee38b
>2012-12-17 09:44:59,144 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] spmStart polling started: taskId = 61a342a5-5d9f-4de2-87bd-56284d45e909
>2012-12-17 09:45:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [33cde09c] Checking autorecoverable hosts
>2012-12-17 09:45:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [33cde09c] Autorecovering 0 hosts
>2012-12-17 09:45:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [33cde09c] Checking autorecoverable hosts done
>2012-12-17 09:45:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [33cde09c] Autorecovering storage domains is disabled, skipping
>2012-12-17 09:45:00,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] spmStart polling ended: taskId = 61a342a5-5d9f-4de2-87bd-56284d45e909 task status = finished
>2012-12-17 09:45:00,186 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] spmStart polling ended, spm status: SPM
>2012-12-17 09:45:00,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=61a342a5-5d9f-4de2-87bd-56284d45e909), log id: 58a9cb0d
>2012-12-17 09:45:00,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] FINISH, HSMClearTaskVDSCommand, log id: 58a9cb0d
>2012-12-17 09:45:00,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7351eff4, log id: 5edee38b
>2012-12-17 09:45:00,217 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6f4b2c75] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:45:00,248 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [21122b75] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:45:00,248 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 55ac2e59
>2012-12-17 09:45:00,274 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6f4b2c75] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:45:00,367 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [16b67351] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:45:00,411 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [16b67351] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1b57bd71
>2012-12-17 09:45:00,413 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [16b67351] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6680e291
>2012-12-17 09:45:00,433 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [16b67351] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:00,569 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [16b67351] FINISH, SpmStopVDSCommand, log id: 6680e291
>2012-12-17 09:45:00,912 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [16b67351] FINISH, ResetIrsVDSCommand, log id: 1b57bd71
>2012-12-17 09:45:00,923 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [16b67351] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6c1abbc2
>2012-12-17 09:45:00,923 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [16b67351] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:00,923 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [16b67351] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6c1abbc2
>2012-12-17 09:45:01,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:45:01,305 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4949
>2012-12-17 09:45:01,307 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4949, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 69e9f0ef
>2012-12-17 09:45:01,328 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] spmStart polling started: taskId = d7aa3cd4-5784-458d-8f97-84d2a5369d1e
>2012-12-17 09:45:02,350 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] spmStart polling ended: taskId = d7aa3cd4-5784-458d-8f97-84d2a5369d1e task status = finished
>2012-12-17 09:45:02,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] spmStart polling ended, spm status: SPM
>2012-12-17 09:45:02,370 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d7aa3cd4-5784-458d-8f97-84d2a5369d1e), log id: 6d0a3c5d
>2012-12-17 09:45:02,388 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, HSMClearTaskVDSCommand, log id: 6d0a3c5d
>2012-12-17 09:45:02,388 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3b013598, log id: 69e9f0ef
>2012-12-17 09:45:02,410 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:45:02,464 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:02,467 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: e46509f
>2012-12-17 09:45:02,488 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: e46509f
>2012-12-17 09:45:02,488 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 55ac2e59
>2012-12-17 09:45:02,488 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [21122b75] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:45:02,488 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2a56fe0b
>2012-12-17 09:45:02,488 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:02,490 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 59d3229c
>2012-12-17 09:45:02,517 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 59d3229c
>2012-12-17 09:45:02,517 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2a56fe0b
>2012-12-17 09:45:02,517 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [21122b75] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:45:03,653 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [622cf65b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:45:11,308 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:45:11,618 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [61b60bdd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:45:11,694 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [61b60bdd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 10c2e81a
>2012-12-17 09:45:11,695 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [61b60bdd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5e3300de
>2012-12-17 09:45:11,715 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [61b60bdd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:11,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [61b60bdd] FINISH, SpmStopVDSCommand, log id: 5e3300de
>2012-12-17 09:45:11,913 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [61b60bdd] FINISH, ResetIrsVDSCommand, log id: 10c2e81a
>2012-12-17 09:45:11,924 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [61b60bdd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 342ceed0
>2012-12-17 09:45:11,924 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [61b60bdd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:11,924 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [61b60bdd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 342ceed0
>2012-12-17 09:45:13,877 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:45:21,986 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:45:22,002 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4950
>2012-12-17 09:45:22,003 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4950, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5fd1d016
>2012-12-17 09:45:22,026 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling started: taskId = 8a974492-f22b-47c3-86e1-e96d4a617a51
>2012-12-17 09:45:23,049 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling ended: taskId = 8a974492-f22b-47c3-86e1-e96d4a617a51 task status = finished
>2012-12-17 09:45:23,068 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling ended, spm status: SPM
>2012-12-17 09:45:23,070 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8a974492-f22b-47c3-86e1-e96d4a617a51), log id: 2afcc4ae
>2012-12-17 09:45:23,089 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, HSMClearTaskVDSCommand, log id: 2afcc4ae
>2012-12-17 09:45:23,089 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@38d4918e, log id: 5fd1d016
>2012-12-17 09:45:23,116 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:45:23,155 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:45:23,155 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5469586d
>2012-12-17 09:45:23,180 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:45:23,270 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [3499e3ae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:45:23,337 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 12ecfacd
>2012-12-17 09:45:23,359 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 67a95c79
>2012-12-17 09:45:23,379 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:23,527 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, SpmStopVDSCommand, log id: 67a95c79
>2012-12-17 09:45:23,538 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, ResetIrsVDSCommand, log id: 12ecfacd
>2012-12-17 09:45:23,549 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2cadcac7
>2012-12-17 09:45:23,549 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3499e3ae] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:23,549 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2cadcac7
>2012-12-17 09:45:23,664 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:45:23,673 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4951
>2012-12-17 09:45:23,675 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4951, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1373e58b
>2012-12-17 09:45:23,698 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling started: taskId = b4b7627c-c1ec-45be-9ad0-9e3f96b7f53b
>2012-12-17 09:45:24,718 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling ended: taskId = b4b7627c-c1ec-45be-9ad0-9e3f96b7f53b task status = finished
>2012-12-17 09:45:24,737 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling ended, spm status: SPM
>2012-12-17 09:45:24,739 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b4b7627c-c1ec-45be-9ad0-9e3f96b7f53b), log id: 62444626
>2012-12-17 09:45:24,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMClearTaskVDSCommand, log id: 62444626
>2012-12-17 09:45:24,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@70599b30, log id: 1373e58b
>2012-12-17 09:45:24,766 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:45:24,783 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:24,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 69670c74
>2012-12-17 09:45:24,803 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 69670c74
>2012-12-17 09:45:24,803 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5469586d
>2012-12-17 09:45:24,803 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:45:24,803 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 49aa4b94
>2012-12-17 09:45:24,803 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:24,805 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1700460d
>2012-12-17 09:45:24,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1700460d
>2012-12-17 09:45:24,823 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 49aa4b94
>2012-12-17 09:45:24,823 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:45:33,703 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:45:33,865 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-22) [6edf3aeb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:45:33,923 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 172de3c3
>2012-12-17 09:45:33,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5f7345ef
>2012-12-17 09:45:33,945 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:34,091 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] FINISH, SpmStopVDSCommand, log id: 5f7345ef
>2012-12-17 09:45:34,136 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] FINISH, ResetIrsVDSCommand, log id: 172de3c3
>2012-12-17 09:45:34,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6157323d
>2012-12-17 09:45:34,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6edf3aeb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:34,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6157323d
>2012-12-17 09:45:34,357 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [24cb4742] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:45:44,217 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [2e298a4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:45:44,337 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [2e298a4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4952
>2012-12-17 09:45:44,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4952, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62873213
>2012-12-17 09:45:44,363 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] spmStart polling started: taskId = ca785d31-0796-4455-b15f-c25ab9721b3a
>2012-12-17 09:45:45,385 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] spmStart polling ended: taskId = ca785d31-0796-4455-b15f-c25ab9721b3a task status = finished
>2012-12-17 09:45:45,403 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] spmStart polling ended, spm status: SPM
>2012-12-17 09:45:45,404 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ca785d31-0796-4455-b15f-c25ab9721b3a), log id: 445bcbbf
>2012-12-17 09:45:45,423 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] FINISH, HSMClearTaskVDSCommand, log id: 445bcbbf
>2012-12-17 09:45:45,423 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@669d8ba8, log id: 62873213
>2012-12-17 09:45:45,472 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [2e298a4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:45:45,584 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:45:45,584 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 462ca497
>2012-12-17 09:45:45,617 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [2e298a4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:45:47,996 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [1dafbc5c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:45:48,129 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [1dafbc5c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 613803f9
>2012-12-17 09:45:48,130 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1dafbc5c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 10449a32
>2012-12-17 09:45:48,153 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1dafbc5c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:48,754 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1dafbc5c] FINISH, SpmStopVDSCommand, log id: 10449a32
>2012-12-17 09:45:48,796 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [1dafbc5c] FINISH, ResetIrsVDSCommand, log id: 613803f9
>2012-12-17 09:45:48,807 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [1dafbc5c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 571e80b7
>2012-12-17 09:45:48,807 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1dafbc5c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:48,807 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [1dafbc5c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 571e80b7
>2012-12-17 09:45:48,928 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:45:48,976 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4953
>2012-12-17 09:45:48,977 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4953, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 58a772ab
>2012-12-17 09:45:49,000 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling started: taskId = db9f1929-2972-46a0-8d5a-aa185ff956c0
>2012-12-17 09:45:50,022 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling ended: taskId = db9f1929-2972-46a0-8d5a-aa185ff956c0 task status = finished
>2012-12-17 09:45:50,041 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling ended, spm status: SPM
>2012-12-17 09:45:50,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=db9f1929-2972-46a0-8d5a-aa185ff956c0), log id: 3338b287
>2012-12-17 09:45:50,061 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMClearTaskVDSCommand, log id: 3338b287
>2012-12-17 09:45:50,061 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f50d756, log id: 58a772ab
>2012-12-17 09:45:50,105 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:45:50,150 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:50,151 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d6331a4
>2012-12-17 09:45:50,171 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d6331a4
>2012-12-17 09:45:50,171 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 462ca497
>2012-12-17 09:45:50,171 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:45:50,171 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1b5035e2
>2012-12-17 09:45:50,171 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:45:50,172 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25d38ecb
>2012-12-17 09:45:50,191 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25d38ecb
>2012-12-17 09:45:50,191 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1b5035e2
>2012-12-17 09:45:50,191 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:45:55,227 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1eb0b032] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:45:58,995 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [48ba699e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:45:59,814 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [12d72057] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:45:59,939 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [12d72057] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1aea4875
>2012-12-17 09:45:59,940 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [12d72057] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: d5eee3e
>2012-12-17 09:45:59,961 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [12d72057] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:00,129 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [12d72057] FINISH, SpmStopVDSCommand, log id: d5eee3e
>2012-12-17 09:46:00,185 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [12d72057] FINISH, ResetIrsVDSCommand, log id: 1aea4875
>2012-12-17 09:46:00,196 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [12d72057] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1d8f7f8e
>2012-12-17 09:46:00,196 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [12d72057] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:00,196 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [12d72057] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1d8f7f8e
>2012-12-17 09:46:05,729 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:46:10,338 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:46:10,443 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4954
>2012-12-17 09:46:10,445 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4954, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e022704
>2012-12-17 09:46:10,469 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] spmStart polling started: taskId = d58652ea-c96b-4bd7-b975-ee8cfca18ba7
>2012-12-17 09:46:11,490 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] spmStart polling ended: taskId = d58652ea-c96b-4bd7-b975-ee8cfca18ba7 task status = finished
>2012-12-17 09:46:11,508 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] spmStart polling ended, spm status: SPM
>2012-12-17 09:46:11,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d58652ea-c96b-4bd7-b975-ee8cfca18ba7), log id: 5e247e4d
>2012-12-17 09:46:11,528 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, HSMClearTaskVDSCommand, log id: 5e247e4d
>2012-12-17 09:46:11,528 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2de02184, log id: 5e022704
>2012-12-17 09:46:11,937 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:46:12,156 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [462eb8a5] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:46:12,157 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 623b96c0
>2012-12-17 09:46:12,195 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:46:12,455 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-92) [565ac2be] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:46:12,585 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5b3bd019
>2012-12-17 09:46:12,587 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 667fd5cb
>2012-12-17 09:46:12,606 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:12,734 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] FINISH, SpmStopVDSCommand, log id: 667fd5cb
>2012-12-17 09:46:12,791 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] FINISH, ResetIrsVDSCommand, log id: 5b3bd019
>2012-12-17 09:46:12,801 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 25f1f07d
>2012-12-17 09:46:12,801 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [565ac2be] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:12,801 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 25f1f07d
>2012-12-17 09:46:12,918 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:46:13,001 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4955
>2012-12-17 09:46:13,002 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4955, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6020028d
>2012-12-17 09:46:13,024 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] spmStart polling started: taskId = 8b6a81bd-f1c1-4c13-bf97-2121e7033276
>2012-12-17 09:46:14,047 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] spmStart polling ended: taskId = 8b6a81bd-f1c1-4c13-bf97-2121e7033276 task status = finished
>2012-12-17 09:46:14,066 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] spmStart polling ended, spm status: SPM
>2012-12-17 09:46:14,067 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8b6a81bd-f1c1-4c13-bf97-2121e7033276), log id: 6be0d07b
>2012-12-17 09:46:14,086 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, HSMClearTaskVDSCommand, log id: 6be0d07b
>2012-12-17 09:46:14,086 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@74977291, log id: 6020028d
>2012-12-17 09:46:14,185 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:46:14,254 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:14,256 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7abbdd7b
>2012-12-17 09:46:14,277 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7abbdd7b
>2012-12-17 09:46:14,277 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 623b96c0
>2012-12-17 09:46:14,277 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [462eb8a5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:46:14,277 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1d217121
>2012-12-17 09:46:14,277 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:14,279 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 59d5be3
>2012-12-17 09:46:14,297 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 59d5be3
>2012-12-17 09:46:14,297 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1d217121
>2012-12-17 09:46:14,297 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [462eb8a5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:46:17,435 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [537845d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:46:22,967 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4f4f0222] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:46:23,233 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-89) [2bcb8475] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:46:23,416 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [2bcb8475] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5ddd4259
>2012-12-17 09:46:23,418 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [2bcb8475] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1dff0cda
>2012-12-17 09:46:23,438 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [2bcb8475] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:23,610 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [2bcb8475] FINISH, SpmStopVDSCommand, log id: 1dff0cda
>2012-12-17 09:46:23,659 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [2bcb8475] FINISH, ResetIrsVDSCommand, log id: 5ddd4259
>2012-12-17 09:46:23,670 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [2bcb8475] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2243934c
>2012-12-17 09:46:23,670 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [2bcb8475] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:23,670 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [2bcb8475] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2243934c
>2012-12-17 09:46:27,855 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:46:33,806 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:46:33,870 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4956
>2012-12-17 09:46:33,871 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4956, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1d791add
>2012-12-17 09:46:33,895 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] spmStart polling started: taskId = 235c9183-bb76-4f8d-a92a-d69d02862812
>2012-12-17 09:46:34,917 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] spmStart polling ended: taskId = 235c9183-bb76-4f8d-a92a-d69d02862812 task status = finished
>2012-12-17 09:46:34,936 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] spmStart polling ended, spm status: SPM
>2012-12-17 09:46:34,937 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=235c9183-bb76-4f8d-a92a-d69d02862812), log id: 1f94cfc1
>2012-12-17 09:46:34,956 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, HSMClearTaskVDSCommand, log id: 1f94cfc1
>2012-12-17 09:46:34,956 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@62d39819, log id: 1d791add
>2012-12-17 09:46:34,986 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:46:35,061 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [42560f9e] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:46:35,061 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5334324f
>2012-12-17 09:46:35,099 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:46:35,338 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [2353b939] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:46:35,379 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: ed578b
>2012-12-17 09:46:35,380 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 176eba74
>2012-12-17 09:46:35,400 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [2353b939] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:35,554 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, SpmStopVDSCommand, log id: 176eba74
>2012-12-17 09:46:35,572 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, ResetIrsVDSCommand, log id: ed578b
>2012-12-17 09:46:35,583 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 957af9d
>2012-12-17 09:46:35,583 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [2353b939] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:35,583 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 957af9d
>2012-12-17 09:46:35,642 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:46:35,655 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4957
>2012-12-17 09:46:35,656 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4957, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 38e4e374
>2012-12-17 09:46:35,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] spmStart polling started: taskId = 4bf882a9-5330-4adc-87a4-29dcc13f641c
>2012-12-17 09:46:36,700 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] spmStart polling ended: taskId = 4bf882a9-5330-4adc-87a4-29dcc13f641c task status = finished
>2012-12-17 09:46:36,719 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] spmStart polling ended, spm status: SPM
>2012-12-17 09:46:36,721 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4bf882a9-5330-4adc-87a4-29dcc13f641c), log id: 53f5e202
>2012-12-17 09:46:36,739 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, HSMClearTaskVDSCommand, log id: 53f5e202
>2012-12-17 09:46:36,739 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@33c91c5e, log id: 38e4e374
>2012-12-17 09:46:36,892 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:46:36,949 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:36,951 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1f506e07
>2012-12-17 09:46:36,970 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1f506e07
>2012-12-17 09:46:36,970 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5334324f
>2012-12-17 09:46:36,970 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [42560f9e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:46:36,970 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 59679da0
>2012-12-17 09:46:36,971 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:36,972 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5a80d926
>2012-12-17 09:46:36,990 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5a80d926
>2012-12-17 09:46:36,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 59679da0
>2012-12-17 09:46:36,990 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [42560f9e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:46:38,193 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:46:45,684 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [61938c83] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:46:47,130 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [2c358b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:46:47,431 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [2c358b5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: e73ecb
>2012-12-17 09:46:47,433 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [2c358b5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3f37644
>2012-12-17 09:46:47,457 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [2c358b5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:47,708 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [2c358b5] FINISH, SpmStopVDSCommand, log id: 3f37644
>2012-12-17 09:46:47,759 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [2c358b5] FINISH, ResetIrsVDSCommand, log id: e73ecb
>2012-12-17 09:46:47,770 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [2c358b5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bb35bc6
>2012-12-17 09:46:47,770 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [2c358b5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:47,770 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [2c358b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bb35bc6
>2012-12-17 09:46:48,418 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [12d72057] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:46:57,843 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [3ad3ad88] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:46:57,868 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [3ad3ad88] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4958
>2012-12-17 09:46:57,870 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4958, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5be7856
>2012-12-17 09:46:57,892 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] spmStart polling started: taskId = 7f537bb5-ea61-4fcd-8164-4bf2e6f04e40
>2012-12-17 09:46:58,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] spmStart polling ended: taskId = 7f537bb5-ea61-4fcd-8164-4bf2e6f04e40 task status = finished
>2012-12-17 09:46:58,932 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] spmStart polling ended, spm status: SPM
>2012-12-17 09:46:58,934 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7f537bb5-ea61-4fcd-8164-4bf2e6f04e40), log id: 4d5d8b04
>2012-12-17 09:46:58,953 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] FINISH, HSMClearTaskVDSCommand, log id: 4d5d8b04
>2012-12-17 09:46:58,953 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3851db4a, log id: 5be7856
>2012-12-17 09:46:58,973 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [3ad3ad88] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:46:59,032 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:46:59,032 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3f7bcdf9
>2012-12-17 09:46:59,060 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [3ad3ad88] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:46:59,192 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-58) [7712cc49] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:46:59,289 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [7712cc49] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6a440a36
>2012-12-17 09:46:59,290 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [7712cc49] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3262893e
>2012-12-17 09:46:59,310 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [7712cc49] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:59,465 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [7712cc49] FINISH, SpmStopVDSCommand, log id: 3262893e
>2012-12-17 09:46:59,491 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [7712cc49] FINISH, ResetIrsVDSCommand, log id: 6a440a36
>2012-12-17 09:46:59,503 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [7712cc49] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3dd658bf
>2012-12-17 09:46:59,503 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7712cc49] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:46:59,503 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [7712cc49] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3dd658bf
>2012-12-17 09:46:59,564 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:46:59,576 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4959
>2012-12-17 09:46:59,577 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4959, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 69bde88
>2012-12-17 09:46:59,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling started: taskId = b172edb7-f8be-44dd-930e-a824187d1255
>2012-12-17 09:47:00,623 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling ended: taskId = b172edb7-f8be-44dd-930e-a824187d1255 task status = finished
>2012-12-17 09:47:00,641 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling ended, spm status: SPM
>2012-12-17 09:47:00,643 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b172edb7-f8be-44dd-930e-a824187d1255), log id: 344e5305
>2012-12-17 09:47:00,669 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMClearTaskVDSCommand, log id: 344e5305
>2012-12-17 09:47:00,669 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2c287adf, log id: 69bde88
>2012-12-17 09:47:00,683 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:47:00,728 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:00,733 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 55acb76c
>2012-12-17 09:47:00,752 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 55acb76c
>2012-12-17 09:47:00,752 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f7bcdf9
>2012-12-17 09:47:00,752 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:47:00,752 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3e3e3f65
>2012-12-17 09:47:00,753 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:00,754 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 389c7c14
>2012-12-17 09:47:00,773 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 389c7c14
>2012-12-17 09:47:00,773 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3e3e3f65
>2012-12-17 09:47:00,773 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:47:08,863 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [78e614fb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:47:09,599 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [e8edb34] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:47:09,838 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [63eed75e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:47:09,998 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3b43e21b
>2012-12-17 09:47:10,000 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 692fa908
>2012-12-17 09:47:10,020 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:10,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] FINISH, SpmStopVDSCommand, log id: 692fa908
>2012-12-17 09:47:10,217 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] FINISH, ResetIrsVDSCommand, log id: 3b43e21b
>2012-12-17 09:47:10,228 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 35e1b6a6
>2012-12-17 09:47:10,228 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [63eed75e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:10,228 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 35e1b6a6
>2012-12-17 09:47:19,182 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:47:20,276 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [31ec2033] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:47:20,291 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [31ec2033] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4960
>2012-12-17 09:47:20,292 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4960, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 500e1121
>2012-12-17 09:47:20,314 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] spmStart polling started: taskId = 26453672-8e03-4815-9229-6aae916a2ebf
>2012-12-17 09:47:21,334 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] spmStart polling ended: taskId = 26453672-8e03-4815-9229-6aae916a2ebf task status = finished
>2012-12-17 09:47:21,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] spmStart polling ended, spm status: SPM
>2012-12-17 09:47:21,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=26453672-8e03-4815-9229-6aae916a2ebf), log id: 76824d1c
>2012-12-17 09:47:21,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] FINISH, HSMClearTaskVDSCommand, log id: 76824d1c
>2012-12-17 09:47:21,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6494a60c, log id: 500e1121
>2012-12-17 09:47:21,686 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [31ec2033] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:47:21,855 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [3e7d3817] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:47:21,856 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 85e9fe9
>2012-12-17 09:47:21,916 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [31ec2033] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:47:22,153 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-35) [53e9685e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:47:22,208 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [53e9685e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 633d492e
>2012-12-17 09:47:22,210 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [53e9685e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: d011071
>2012-12-17 09:47:22,230 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [53e9685e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:22,582 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [53e9685e] FINISH, SpmStopVDSCommand, log id: d011071
>2012-12-17 09:47:22,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [53e9685e] FINISH, ResetIrsVDSCommand, log id: 633d492e
>2012-12-17 09:47:22,610 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [53e9685e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 307cfba5
>2012-12-17 09:47:22,610 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [53e9685e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:22,610 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [53e9685e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 307cfba5
>2012-12-17 09:47:22,668 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:47:22,681 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4961
>2012-12-17 09:47:22,682 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4961, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4f005956
>2012-12-17 09:47:22,704 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling started: taskId = 1d4eab4e-a549-4dff-8f49-db997b01ab55
>2012-12-17 09:47:23,726 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling ended: taskId = 1d4eab4e-a549-4dff-8f49-db997b01ab55 task status = finished
>2012-12-17 09:47:23,744 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling ended, spm status: SPM
>2012-12-17 09:47:23,745 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1d4eab4e-a549-4dff-8f49-db997b01ab55), log id: cbdc0a0
>2012-12-17 09:47:23,764 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, HSMClearTaskVDSCommand, log id: cbdc0a0
>2012-12-17 09:47:23,764 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7db14ee8, log id: 4f005956
>2012-12-17 09:47:23,789 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:47:23,819 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:23,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5b5fdd5e
>2012-12-17 09:47:23,843 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5b5fdd5e
>2012-12-17 09:47:23,843 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 85e9fe9
>2012-12-17 09:47:23,843 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [3e7d3817] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:47:23,843 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 72be752f
>2012-12-17 09:47:23,843 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:23,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5f03daa3
>2012-12-17 09:47:23,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5f03daa3
>2012-12-17 09:47:23,864 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 72be752f
>2012-12-17 09:47:23,864 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [3e7d3817] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:47:29,488 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [680acadd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:47:32,708 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [603633ae] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:47:32,842 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [2b52624] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:47:32,903 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [2b52624] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 30354684
>2012-12-17 09:47:32,905 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [2b52624] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2951bb72
>2012-12-17 09:47:32,925 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [2b52624] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:33,413 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [2b52624] FINISH, SpmStopVDSCommand, log id: 2951bb72
>2012-12-17 09:47:33,469 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [2b52624] FINISH, ResetIrsVDSCommand, log id: 30354684
>2012-12-17 09:47:33,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [2b52624] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c029617
>2012-12-17 09:47:33,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2b52624] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:33,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [2b52624] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c029617
>2012-12-17 09:47:39,708 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [2420ee59] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:47:43,606 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:47:43,633 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4962
>2012-12-17 09:47:43,635 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4962, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1f762c0
>2012-12-17 09:47:43,656 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling started: taskId = 95d187dc-0da9-44ca-b393-8ed5a996d838
>2012-12-17 09:47:44,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling ended: taskId = 95d187dc-0da9-44ca-b393-8ed5a996d838 task status = finished
>2012-12-17 09:47:44,696 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling ended, spm status: SPM
>2012-12-17 09:47:44,697 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=95d187dc-0da9-44ca-b393-8ed5a996d838), log id: 418fa6eb
>2012-12-17 09:47:44,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, HSMClearTaskVDSCommand, log id: 418fa6eb
>2012-12-17 09:47:44,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e17afed, log id: 1f762c0
>2012-12-17 09:47:44,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:47:44,794 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [7646beaa] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:47:44,794 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6f36ac24
>2012-12-17 09:47:44,852 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:47:44,944 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-11) [69818e3e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:47:44,995 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-11) [69818e3e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4fbbe20f
>2012-12-17 09:47:44,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-11) [69818e3e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4b18c042
>2012-12-17 09:47:45,017 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-11) [69818e3e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:45,175 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-11) [69818e3e] FINISH, SpmStopVDSCommand, log id: 4b18c042
>2012-12-17 09:47:45,192 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-11) [69818e3e] FINISH, ResetIrsVDSCommand, log id: 4fbbe20f
>2012-12-17 09:47:45,203 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-11) [69818e3e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 79da9f0d
>2012-12-17 09:47:45,203 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [69818e3e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:45,203 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-11) [69818e3e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 79da9f0d
>2012-12-17 09:47:45,278 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:47:45,293 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4963
>2012-12-17 09:47:45,294 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4963, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2dbf3ca7
>2012-12-17 09:47:45,316 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] spmStart polling started: taskId = c7f0b919-0df8-4508-904c-a8d2ecbdd29a
>2012-12-17 09:47:46,341 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] spmStart polling ended: taskId = c7f0b919-0df8-4508-904c-a8d2ecbdd29a task status = finished
>2012-12-17 09:47:46,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] spmStart polling ended, spm status: SPM
>2012-12-17 09:47:46,364 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c7f0b919-0df8-4508-904c-a8d2ecbdd29a), log id: 5321103b
>2012-12-17 09:47:46,383 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, HSMClearTaskVDSCommand, log id: 5321103b
>2012-12-17 09:47:46,383 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1fb0bc20, log id: 2dbf3ca7
>2012-12-17 09:47:46,434 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:47:46,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:46,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: f6d57a9
>2012-12-17 09:47:46,499 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: f6d57a9
>2012-12-17 09:47:46,500 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6f36ac24
>2012-12-17 09:47:46,500 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [7646beaa] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:47:46,500 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 13ff2255
>2012-12-17 09:47:46,500 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:46,501 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: ee160e
>2012-12-17 09:47:46,519 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: ee160e
>2012-12-17 09:47:46,520 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 13ff2255
>2012-12-17 09:47:46,520 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [7646beaa] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:47:49,945 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:47:55,365 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [4d6b231f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:47:55,554 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:47:55,665 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2577306
>2012-12-17 09:47:55,667 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26ec39f3
>2012-12-17 09:47:55,686 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:55,838 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] FINISH, SpmStopVDSCommand, log id: 26ec39f3
>2012-12-17 09:47:56,678 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] FINISH, ResetIrsVDSCommand, log id: 2577306
>2012-12-17 09:47:56,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7fb360cb
>2012-12-17 09:47:56,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:47:56,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7fb360cb
>2012-12-17 09:48:00,185 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [24cb4742] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:48:07,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:48:07,061 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4964
>2012-12-17 09:48:07,062 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4964, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67ff327b
>2012-12-17 09:48:07,084 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] spmStart polling started: taskId = 02c72871-f2d0-467a-9113-3d9a56b7246e
>2012-12-17 09:48:08,106 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] spmStart polling ended: taskId = 02c72871-f2d0-467a-9113-3d9a56b7246e task status = finished
>2012-12-17 09:48:08,124 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] spmStart polling ended, spm status: SPM
>2012-12-17 09:48:08,126 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=02c72871-f2d0-467a-9113-3d9a56b7246e), log id: 6c36358e
>2012-12-17 09:48:08,144 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] FINISH, HSMClearTaskVDSCommand, log id: 6c36358e
>2012-12-17 09:48:08,144 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5af63522, log id: 67ff327b
>2012-12-17 09:48:08,156 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:48:08,187 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [515f648d] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:48:08,187 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 70a78f58
>2012-12-17 09:48:08,217 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:48:08,393 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [3d5285e1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:48:08,433 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1d10621
>2012-12-17 09:48:08,435 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2bb0b606
>2012-12-17 09:48:08,454 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:09,066 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, SpmStopVDSCommand, log id: 2bb0b606
>2012-12-17 09:48:09,084 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, ResetIrsVDSCommand, log id: 1d10621
>2012-12-17 09:48:09,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 519e3c94
>2012-12-17 09:48:09,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3d5285e1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:09,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 519e3c94
>2012-12-17 09:48:09,139 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [515f648d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:48:09,148 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [515f648d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4965
>2012-12-17 09:48:09,150 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4965, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 52abcef9
>2012-12-17 09:48:09,173 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [515f648d] spmStart polling started: taskId = 1e24d303-bbfe-474b-86d4-022bac2afb3c
>2012-12-17 09:48:10,196 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [515f648d] spmStart polling ended: taskId = 1e24d303-bbfe-474b-86d4-022bac2afb3c task status = finished
>2012-12-17 09:48:10,215 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [515f648d] spmStart polling ended, spm status: SPM
>2012-12-17 09:48:10,216 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1e24d303-bbfe-474b-86d4-022bac2afb3c), log id: 59aa1661
>2012-12-17 09:48:10,235 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, HSMClearTaskVDSCommand, log id: 59aa1661
>2012-12-17 09:48:10,235 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2da36844, log id: 52abcef9
>2012-12-17 09:48:10,248 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [515f648d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:48:10,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:10,273 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d69d974
>2012-12-17 09:48:10,293 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d69d974
>2012-12-17 09:48:10,293 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 70a78f58
>2012-12-17 09:48:10,293 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [515f648d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:48:10,293 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 404386ca
>2012-12-17 09:48:10,293 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:10,296 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 182129c1
>2012-12-17 09:48:10,336 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 182129c1
>2012-12-17 09:48:10,336 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 404386ca
>2012-12-17 09:48:10,336 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [515f648d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:48:10,425 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [2353b939] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:48:19,181 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [7781d588] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:48:19,325 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [469720a2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:48:19,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [469720a2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 48f91495
>2012-12-17 09:48:19,370 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [469720a2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 75975b4f
>2012-12-17 09:48:19,390 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [469720a2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:19,587 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [469720a2] FINISH, SpmStopVDSCommand, log id: 75975b4f
>2012-12-17 09:48:19,685 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [469720a2] FINISH, ResetIrsVDSCommand, log id: 48f91495
>2012-12-17 09:48:19,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [469720a2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 61a7eee9
>2012-12-17 09:48:19,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [469720a2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:19,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [469720a2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 61a7eee9
>2012-12-17 09:48:20,601 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:48:29,763 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:48:29,798 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4966
>2012-12-17 09:48:29,800 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4966, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 32bc1a2d
>2012-12-17 09:48:29,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] spmStart polling started: taskId = 7e06ddf1-4d36-4ecd-9d80-323272d41594
>2012-12-17 09:48:30,844 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] spmStart polling ended: taskId = 7e06ddf1-4d36-4ecd-9d80-323272d41594 task status = finished
>2012-12-17 09:48:30,867 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] spmStart polling ended, spm status: SPM
>2012-12-17 09:48:30,868 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7e06ddf1-4d36-4ecd-9d80-323272d41594), log id: 5b074175
>2012-12-17 09:48:30,888 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, HSMClearTaskVDSCommand, log id: 5b074175
>2012-12-17 09:48:30,888 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@34153ba7, log id: 32bc1a2d
>2012-12-17 09:48:31,538 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:48:31,738 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:48:31,739 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3dbea72
>2012-12-17 09:48:31,750 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:48:31,902 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [13c8ba9c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:48:31,955 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7c25fa0d
>2012-12-17 09:48:31,957 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 381760d5
>2012-12-17 09:48:31,978 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:32,133 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, SpmStopVDSCommand, log id: 381760d5
>2012-12-17 09:48:32,155 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, ResetIrsVDSCommand, log id: 7c25fa0d
>2012-12-17 09:48:32,167 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 228a71b4
>2012-12-17 09:48:32,167 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:32,167 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 228a71b4
>2012-12-17 09:48:32,261 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:48:32,280 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4967
>2012-12-17 09:48:32,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4967, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 610a2666
>2012-12-17 09:48:32,305 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling started: taskId = 5ea942f9-9608-4d4f-abd4-1b7d89710683
>2012-12-17 09:48:33,330 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling ended: taskId = 5ea942f9-9608-4d4f-abd4-1b7d89710683 task status = finished
>2012-12-17 09:48:33,350 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling ended, spm status: SPM
>2012-12-17 09:48:33,352 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5ea942f9-9608-4d4f-abd4-1b7d89710683), log id: 4ddcaf9f
>2012-12-17 09:48:33,371 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMClearTaskVDSCommand, log id: 4ddcaf9f
>2012-12-17 09:48:33,371 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a813717, log id: 610a2666
>2012-12-17 09:48:33,392 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:48:33,422 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:33,426 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 38bbd0a9
>2012-12-17 09:48:33,446 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 38bbd0a9
>2012-12-17 09:48:33,446 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3dbea72
>2012-12-17 09:48:33,446 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:48:33,446 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7ad2274e
>2012-12-17 09:48:33,446 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:33,448 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 785fad50
>2012-12-17 09:48:33,467 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 785fad50
>2012-12-17 09:48:33,467 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7ad2274e
>2012-12-17 09:48:33,467 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:48:42,028 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:48:42,305 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:48:42,518 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [49d29b47] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:48:42,582 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [49d29b47] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 684337fe
>2012-12-17 09:48:42,583 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [49d29b47] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 60fa540b
>2012-12-17 09:48:42,603 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [49d29b47] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:42,794 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [49d29b47] FINISH, SpmStopVDSCommand, log id: 60fa540b
>2012-12-17 09:48:42,817 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [49d29b47] FINISH, ResetIrsVDSCommand, log id: 684337fe
>2012-12-17 09:48:42,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [49d29b47] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2ae86858
>2012-12-17 09:48:42,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [49d29b47] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:42,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [49d29b47] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2ae86858
>2012-12-17 09:48:52,209 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:48:52,893 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:48:52,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4968
>2012-12-17 09:48:52,923 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4968, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7838e427
>2012-12-17 09:48:52,951 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling started: taskId = 72a8c156-96d2-4e80-a71a-0f61f23ecd82
>2012-12-17 09:48:53,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling ended: taskId = 72a8c156-96d2-4e80-a71a-0f61f23ecd82 task status = finished
>2012-12-17 09:48:53,993 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling ended, spm status: SPM
>2012-12-17 09:48:53,994 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=72a8c156-96d2-4e80-a71a-0f61f23ecd82), log id: 60e0d390
>2012-12-17 09:48:54,013 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, HSMClearTaskVDSCommand, log id: 60e0d390
>2012-12-17 09:48:54,013 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4861c4a0, log id: 7838e427
>2012-12-17 09:48:54,026 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:48:54,071 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:48:54,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5c410c4a
>2012-12-17 09:48:54,092 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:48:54,232 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-79) [9d8cf45] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:48:54,283 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: c91a391
>2012-12-17 09:48:54,284 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: b37cb5f
>2012-12-17 09:48:54,304 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:54,793 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] FINISH, SpmStopVDSCommand, log id: b37cb5f
>2012-12-17 09:48:54,812 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] FINISH, ResetIrsVDSCommand, log id: c91a391
>2012-12-17 09:48:54,823 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2a82e070
>2012-12-17 09:48:54,823 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [9d8cf45] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:54,823 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2a82e070
>2012-12-17 09:48:54,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:48:54,925 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4969
>2012-12-17 09:48:54,926 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4969, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: dc5c9ad
>2012-12-17 09:48:54,949 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling started: taskId = b4bc3b5d-6605-4dba-b706-ceb2fa52b1d7
>2012-12-17 09:48:55,972 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended: taskId = b4bc3b5d-6605-4dba-b706-ceb2fa52b1d7 task status = finished
>2012-12-17 09:48:55,990 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended, spm status: SPM
>2012-12-17 09:48:55,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b4bc3b5d-6605-4dba-b706-ceb2fa52b1d7), log id: 71ab63ea
>2012-12-17 09:48:56,010 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMClearTaskVDSCommand, log id: 71ab63ea
>2012-12-17 09:48:56,010 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2b2cd1f3, log id: dc5c9ad
>2012-12-17 09:48:56,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:48:56,434 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:56,439 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: dc5ee60
>2012-12-17 09:48:56,460 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: dc5ee60
>2012-12-17 09:48:56,460 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5c410c4a
>2012-12-17 09:48:56,460 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:48:56,460 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3da4e7b0
>2012-12-17 09:48:56,460 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:48:56,462 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 51c04146
>2012-12-17 09:48:56,481 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 51c04146
>2012-12-17 09:48:56,481 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3da4e7b0
>2012-12-17 09:48:56,481 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:49:02,428 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:49:04,938 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [680acadd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:49:05,090 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-29) [2b3eea84] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:49:05,145 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 671c05fc
>2012-12-17 09:49:05,146 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3578ebbc
>2012-12-17 09:49:05,166 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:49:05,802 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, SpmStopVDSCommand, log id: 3578ebbc
>2012-12-17 09:49:06,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, ResetIrsVDSCommand, log id: 671c05fc
>2012-12-17 09:49:06,299 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e552012
>2012-12-17 09:49:06,299 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [2b3eea84] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:49:06,299 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e552012
>2012-12-17 09:49:12,652 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [2c358b5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:49:16,389 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:49:16,417 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4970
>2012-12-17 09:49:16,418 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4970, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7aa26cf6
>2012-12-17 09:49:16,441 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] spmStart polling started: taskId = aa5ded6e-55d8-42b9-b6a1-ac14fa5fa528
>2012-12-17 09:49:17,462 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] spmStart polling ended: taskId = aa5ded6e-55d8-42b9-b6a1-ac14fa5fa528 task status = finished
>2012-12-17 09:49:17,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] spmStart polling ended, spm status: SPM
>2012-12-17 09:49:17,481 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=aa5ded6e-55d8-42b9-b6a1-ac14fa5fa528), log id: 5f65b4b6
>2012-12-17 09:49:17,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, HSMClearTaskVDSCommand, log id: 5f65b4b6
>2012-12-17 09:49:17,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e3b49a8, log id: 7aa26cf6
>2012-12-17 09:49:17,511 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:49:17,545 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [56a7c4a] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:49:17,545 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 438204a8
>2012-12-17 09:49:17,573 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:49:17,660 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-54) [510758b1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:49:17,717 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [510758b1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6df764f2
>2012-12-17 09:49:17,718 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [510758b1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2fcf2b3
>2012-12-17 09:49:17,738 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [510758b1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:49:17,877 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [510758b1] FINISH, SpmStopVDSCommand, log id: 2fcf2b3
>2012-12-17 09:49:17,895 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [510758b1] FINISH, ResetIrsVDSCommand, log id: 6df764f2
>2012-12-17 09:49:17,906 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [510758b1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e67df6c
>2012-12-17 09:49:17,906 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [510758b1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:49:17,906 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [510758b1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e67df6c
>2012-12-17 09:49:17,953 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:49:17,970 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4971
>2012-12-17 09:49:17,971 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4971, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2644c0e7
>2012-12-17 09:49:17,993 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] spmStart polling started: taskId = 2172aef2-2d65-46ca-8e02-20cd56b19bee
>2012-12-17 09:49:19,014 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] spmStart polling ended: taskId = 2172aef2-2d65-46ca-8e02-20cd56b19bee task status = finished
>2012-12-17 09:49:19,033 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] spmStart polling ended, spm status: SPM
>2012-12-17 09:49:19,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2172aef2-2d65-46ca-8e02-20cd56b19bee), log id: 5943be32
>2012-12-17 09:49:19,056 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, HSMClearTaskVDSCommand, log id: 5943be32
>2012-12-17 09:49:19,056 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6e7bf266, log id: 2644c0e7
>2012-12-17 09:49:19,100 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:49:19,113 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:49:19,117 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 19049ecb
>2012-12-17 09:49:19,137 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 19049ecb
>2012-12-17 09:49:19,137 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 438204a8
>2012-12-17 09:49:19,137 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [56a7c4a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:49:19,137 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: a4fee33
>2012-12-17 09:49:19,137 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:49:19,138 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4abd8f52
>2012-12-17 09:49:19,157 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4abd8f52
>2012-12-17 09:49:19,157 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a4fee33
>2012-12-17 09:49:19,157 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [56a7c4a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:49:22,853 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:49:28,000 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [65ba9edd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:49:28,136 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-56) [4bbb254] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:49:28,185 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [4bbb254] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 61bce2b0
>2012-12-17 09:49:28,186 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [4bbb254] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5a2186ff
>2012-12-17 09:49:28,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [4bbb254] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:49:28,376 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [4bbb254] FINISH, SpmStopVDSCommand, log id: 5a2186ff
>2012-12-17 09:49:28,396 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [4bbb254] FINISH, ResetIrsVDSCommand, log id: 61bce2b0
>2012-12-17 09:49:28,406 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [4bbb254] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 340e73ba
>2012-12-17 09:49:28,406 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [4bbb254] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:49:28,406 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [4bbb254] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 340e73ba
>2012-12-17 09:49:33,091 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:49:38,463 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [58bd42f7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:49:38,544 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [58bd42f7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4972
>2012-12-17 09:49:38,546 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4972, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7bb857ed
>2012-12-17 09:49:38,568 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] spmStart polling started: taskId = 7a86b9e4-ee9b-4f79-be93-b35e18102e0e
>2012-12-17 09:49:39,589 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] spmStart polling ended: taskId = 7a86b9e4-ee9b-4f79-be93-b35e18102e0e task status = finished
>2012-12-17 09:49:39,608 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] spmStart polling ended, spm status: SPM
>2012-12-17 09:49:39,609 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7a86b9e4-ee9b-4f79-be93-b35e18102e0e), log id: 6709929b
>2012-12-17 09:49:39,627 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] FINISH, HSMClearTaskVDSCommand, log id: 6709929b
>2012-12-17 09:49:39,627 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@257832f6, log id: 7bb857ed
>2012-12-17 09:49:39,705 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [58bd42f7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:49:39,908 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [75733504] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:49:39,908 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6ef4a17b
>2012-12-17 09:49:39,908 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [58bd42f7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:49:40,110 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-36) [38047437] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:49:40,178 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [38047437] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7c215e40
>2012-12-17 09:49:40,179 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [38047437] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 35021459
>2012-12-17 09:49:40,199 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [38047437] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:49:40,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [38047437] FINISH, SpmStopVDSCommand, log id: 35021459
>2012-12-17 09:49:40,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [38047437] FINISH, ResetIrsVDSCommand, log id: 7c215e40
>2012-12-17 09:49:40,710 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [38047437] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e1c52dd
>2012-12-17 09:49:40,710 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [38047437] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:49:40,710 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [38047437] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e1c52dd
>2012-12-17 09:49:40,776 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:49:41,412 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4973
>2012-12-17 09:49:41,414 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4973, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b822200
>2012-12-17 09:49:41,436 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] spmStart polling started: taskId = 46b1d296-4158-41a4-9ac2-1d2468c3326f
>2012-12-17 09:49:42,458 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] spmStart polling ended: taskId = 46b1d296-4158-41a4-9ac2-1d2468c3326f task status = finished
>2012-12-17 09:49:42,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] spmStart polling ended, spm status: SPM
>2012-12-17 09:49:42,478 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=46b1d296-4158-41a4-9ac2-1d2468c3326f), log id: 30e94a07
>2012-12-17 09:49:42,496 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, HSMClearTaskVDSCommand, log id: 30e94a07
>2012-12-17 09:49:42,496 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@23fb2aa7, log id: 6b822200
>2012-12-17 09:49:42,515 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:49:42,533 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:49:42,534 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2a815e1d
>2012-12-17 09:49:42,554 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2a815e1d
>2012-12-17 09:49:42,554 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6ef4a17b
>2012-12-17 09:49:42,555 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [75733504] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:49:42,555 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1349d7b8
>2012-12-17 09:49:42,555 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:49:42,556 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25ec675e
>2012-12-17 09:49:42,575 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25ec675e
>2012-12-17 09:49:42,575 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1349d7b8
>2012-12-17 09:49:42,575 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [75733504] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:49:43,650 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [53e9685e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:49:51,394 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [130592b9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:49:51,513 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [312ad09c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:49:51,563 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4182e797
>2012-12-17 09:49:51,565 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 275da8bd
>2012-12-17 09:49:51,584 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:49:51,836 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, SpmStopVDSCommand, log id: 275da8bd
>2012-12-17 09:49:51,849 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, ResetIrsVDSCommand, log id: 4182e797
>2012-12-17 09:49:51,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 567f91e8
>2012-12-17 09:49:51,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:49:51,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 567f91e8
>2012-12-17 09:49:53,837 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [2b3eea84] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:50:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [12d72057] Checking autorecoverable hosts
>2012-12-17 09:50:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [12d72057] Autorecovering 0 hosts
>2012-12-17 09:50:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [12d72057] Checking autorecoverable hosts done
>2012-12-17 09:50:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [12d72057] Autorecovering storage domains is disabled, skipping
>2012-12-17 09:50:01,926 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [519a0bb9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:50:04,150 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [519a0bb9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4974
>2012-12-17 09:50:04,152 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4974, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 487299e4
>2012-12-17 09:50:04,175 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] spmStart polling started: taskId = 03efe095-7f88-49fa-a5c3-e1a0c05d7efa
>2012-12-17 09:50:05,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] spmStart polling ended: taskId = 03efe095-7f88-49fa-a5c3-e1a0c05d7efa task status = finished
>2012-12-17 09:50:05,214 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] spmStart polling ended, spm status: SPM
>2012-12-17 09:50:05,216 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=03efe095-7f88-49fa-a5c3-e1a0c05d7efa), log id: 4de126b3
>2012-12-17 09:50:05,235 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] FINISH, HSMClearTaskVDSCommand, log id: 4de126b3
>2012-12-17 09:50:05,235 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78b3d9b6, log id: 487299e4
>2012-12-17 09:50:05,247 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [519a0bb9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:50:05,284 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [13c8ba9c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:50:05,284 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2df299ca
>2012-12-17 09:50:05,320 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [519a0bb9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:50:05,464 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [2111ab39] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:50:05,554 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 45276ffc
>2012-12-17 09:50:05,556 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5b38696b
>2012-12-17 09:50:05,575 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:50:06,283 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] FINISH, SpmStopVDSCommand, log id: 5b38696b
>2012-12-17 09:50:06,790 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] FINISH, ResetIrsVDSCommand, log id: 45276ffc
>2012-12-17 09:50:06,801 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24803349
>2012-12-17 09:50:06,801 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2111ab39] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:50:06,802 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24803349
>2012-12-17 09:50:06,883 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:50:06,895 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4975
>2012-12-17 09:50:06,897 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4975, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: e0ee927
>2012-12-17 09:50:06,919 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] spmStart polling started: taskId = 570eccbd-1961-475c-aefe-8c719d723668
>2012-12-17 09:50:07,941 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] spmStart polling ended: taskId = 570eccbd-1961-475c-aefe-8c719d723668 task status = finished
>2012-12-17 09:50:07,961 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] spmStart polling ended, spm status: SPM
>2012-12-17 09:50:07,962 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=570eccbd-1961-475c-aefe-8c719d723668), log id: 5fbf22b9
>2012-12-17 09:50:07,981 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, HSMClearTaskVDSCommand, log id: 5fbf22b9
>2012-12-17 09:50:07,981 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7d80d0ca, log id: e0ee927
>2012-12-17 09:50:08,002 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:50:08,022 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:50:08,026 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 181203e3
>2012-12-17 09:50:08,047 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 181203e3
>2012-12-17 09:50:08,047 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2df299ca
>2012-12-17 09:50:08,047 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [13c8ba9c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:50:08,047 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 46bd386d
>2012-12-17 09:50:08,047 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:50:08,049 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 33ec1547
>2012-12-17 09:50:08,067 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 33ec1547
>2012-12-17 09:50:08,068 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 46bd386d
>2012-12-17 09:50:08,068 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [13c8ba9c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:50:14,649 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:50:16,921 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [49d29b47] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:50:17,042 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [3e0f3441] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:50:17,103 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1b26b838
>2012-12-17 09:50:17,105 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26618845
>2012-12-17 09:50:17,124 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:50:17,273 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, SpmStopVDSCommand, log id: 26618845
>2012-12-17 09:50:17,289 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, ResetIrsVDSCommand, log id: 1b26b838
>2012-12-17 09:50:17,299 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 267254c3
>2012-12-17 09:50:17,300 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:50:17,300 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 267254c3
>2012-12-17 09:50:25,203 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:50:27,360 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [be98275] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:50:27,412 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [be98275] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4976
>2012-12-17 09:50:27,415 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [be98275] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4976, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5631fdc9
>2012-12-17 09:50:27,437 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [be98275] spmStart polling started: taskId = dc8b7586-3c66-46a8-a8a6-19fa64aef5cb
>2012-12-17 09:50:28,460 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [be98275] spmStart polling ended: taskId = dc8b7586-3c66-46a8-a8a6-19fa64aef5cb task status = finished
>2012-12-17 09:50:28,479 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [be98275] spmStart polling ended, spm status: SPM
>2012-12-17 09:50:28,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [be98275] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dc8b7586-3c66-46a8-a8a6-19fa64aef5cb), log id: 1e7d9cb3
>2012-12-17 09:50:28,498 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [be98275] FINISH, HSMClearTaskVDSCommand, log id: 1e7d9cb3
>2012-12-17 09:50:28,498 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [be98275] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48babb68, log id: 5631fdc9
>2012-12-17 09:50:28,654 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [be98275] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:50:28,802 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [4ea30a59] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:50:28,802 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 338ea258
>2012-12-17 09:50:28,849 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [be98275] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:50:29,296 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [5e506f3b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:50:30,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3390c1b4
>2012-12-17 09:50:30,376 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 538d09eb
>2012-12-17 09:50:30,396 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:50:30,539 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] FINISH, SpmStopVDSCommand, log id: 538d09eb
>2012-12-17 09:50:31,317 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] FINISH, ResetIrsVDSCommand, log id: 3390c1b4
>2012-12-17 09:50:31,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6c967043
>2012-12-17 09:50:31,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5e506f3b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:50:31,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6c967043
>2012-12-17 09:50:33,283 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:50:33,337 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4977
>2012-12-17 09:50:33,339 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4977, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4b4efdc
>2012-12-17 09:50:33,390 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] spmStart polling started: taskId = 141603a7-d0f8-41f9-bb3a-30ada5c07a1f
>2012-12-17 09:50:34,412 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] spmStart polling ended: taskId = 141603a7-d0f8-41f9-bb3a-30ada5c07a1f task status = finished
>2012-12-17 09:50:34,431 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] spmStart polling ended, spm status: SPM
>2012-12-17 09:50:34,433 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=141603a7-d0f8-41f9-bb3a-30ada5c07a1f), log id: 3f47b1b1
>2012-12-17 09:50:34,451 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, HSMClearTaskVDSCommand, log id: 3f47b1b1
>2012-12-17 09:50:34,451 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7be2fbeb, log id: 4b4efdc
>2012-12-17 09:50:34,489 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:50:34,506 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:50:34,508 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 11d2420c
>2012-12-17 09:50:34,527 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 11d2420c
>2012-12-17 09:50:34,527 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 338ea258
>2012-12-17 09:50:34,527 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [4ea30a59] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:50:34,527 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 62470148
>2012-12-17 09:50:34,528 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:50:34,529 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3bc51bd6
>2012-12-17 09:50:34,548 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3bc51bd6
>2012-12-17 09:50:34,548 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 62470148
>2012-12-17 09:50:34,548 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [4ea30a59] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:50:35,542 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:50:43,842 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [278943a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:50:44,433 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [7fd67d3e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:50:44,750 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [7fd67d3e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4f9bdcc1
>2012-12-17 09:50:44,752 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [7fd67d3e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 45b52c72
>2012-12-17 09:50:44,773 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [7fd67d3e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:50:44,940 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [7fd67d3e] FINISH, SpmStopVDSCommand, log id: 45b52c72
>2012-12-17 09:50:45,023 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [7fd67d3e] FINISH, ResetIrsVDSCommand, log id: 4f9bdcc1
>2012-12-17 09:50:45,034 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [7fd67d3e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 482e2e92
>2012-12-17 09:50:45,034 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [7fd67d3e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:50:45,034 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [7fd67d3e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 482e2e92
>2012-12-17 09:50:47,027 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2b52624] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:50:55,268 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7fbbf72c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:50:55,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7fbbf72c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4978
>2012-12-17 09:50:55,291 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4978, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 71fcbe3
>2012-12-17 09:50:55,315 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] spmStart polling started: taskId = ed15f365-6f0c-4983-8a2f-e34c91774a17
>2012-12-17 09:50:56,336 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] spmStart polling ended: taskId = ed15f365-6f0c-4983-8a2f-e34c91774a17 task status = finished
>2012-12-17 09:50:56,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] spmStart polling ended, spm status: SPM
>2012-12-17 09:50:56,355 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ed15f365-6f0c-4983-8a2f-e34c91774a17), log id: 48e03ce3
>2012-12-17 09:50:56,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] FINISH, HSMClearTaskVDSCommand, log id: 48e03ce3
>2012-12-17 09:50:56,374 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@79bd8075, log id: 71fcbe3
>2012-12-17 09:50:56,863 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7fbbf72c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:50:57,440 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-19) [7a6ed577] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:50:57,440 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: e2ab496
>2012-12-17 09:50:57,468 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7fbbf72c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:50:57,906 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [477daf1e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:50:59,091 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 24bdfadb
>2012-12-17 09:50:59,093 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5937ffac
>2012-12-17 09:50:59,113 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:50:59,270 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, SpmStopVDSCommand, log id: 5937ffac
>2012-12-17 09:50:59,611 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [69818e3e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:51:00,259 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, ResetIrsVDSCommand, log id: 24bdfadb
>2012-12-17 09:51:00,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4340bfd
>2012-12-17 09:51:00,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [477daf1e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:00,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4340bfd
>2012-12-17 09:51:03,750 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:51:03,865 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4979
>2012-12-17 09:51:03,867 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4979, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 28aeda01
>2012-12-17 09:51:03,889 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] spmStart polling started: taskId = 45cf1767-48e9-4dba-a2ab-b42afc9a8edc
>2012-12-17 09:51:04,911 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] spmStart polling ended: taskId = 45cf1767-48e9-4dba-a2ab-b42afc9a8edc task status = finished
>2012-12-17 09:51:04,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] spmStart polling ended, spm status: SPM
>2012-12-17 09:51:04,931 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=45cf1767-48e9-4dba-a2ab-b42afc9a8edc), log id: 44493f14
>2012-12-17 09:51:04,950 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, HSMClearTaskVDSCommand, log id: 44493f14
>2012-12-17 09:51:04,950 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@24cc5d12, log id: 28aeda01
>2012-12-17 09:51:04,997 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:51:05,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:05,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7ff80de4
>2012-12-17 09:51:05,070 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7ff80de4
>2012-12-17 09:51:05,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: e2ab496
>2012-12-17 09:51:05,071 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-19) [7a6ed577] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:51:05,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7658355e
>2012-12-17 09:51:05,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:05,072 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 34795f86
>2012-12-17 09:51:05,091 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 34795f86
>2012-12-17 09:51:05,091 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7658355e
>2012-12-17 09:51:05,091 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-19) [7a6ed577] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:51:05,137 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:51:05,137 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 26d2482c
>2012-12-17 09:51:05,137 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:05,139 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6f45ee5f
>2012-12-17 09:51:05,157 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f45ee5f
>2012-12-17 09:51:05,157 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 26d2482c
>2012-12-17 09:51:05,157 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:51:05,157 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5824d80b
>2012-12-17 09:51:05,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:05,159 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1450ee3d
>2012-12-17 09:51:05,177 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1450ee3d
>2012-12-17 09:51:05,177 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5824d80b
>2012-12-17 09:51:05,177 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:51:13,845 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [632afe2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:51:14,700 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [23663da4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:51:14,884 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [23663da4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6879d3e0
>2012-12-17 09:51:14,886 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [23663da4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6bba85a1
>2012-12-17 09:51:14,906 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [23663da4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:15,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [23663da4] FINISH, SpmStopVDSCommand, log id: 6bba85a1
>2012-12-17 09:51:15,193 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [23663da4] FINISH, ResetIrsVDSCommand, log id: 6879d3e0
>2012-12-17 09:51:15,211 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [23663da4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e65d6c3
>2012-12-17 09:51:15,211 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [23663da4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:15,211 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [23663da4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e65d6c3
>2012-12-17 09:51:23,392 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:51:25,334 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:51:25,360 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4980
>2012-12-17 09:51:25,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4980, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5265faaf
>2012-12-17 09:51:25,385 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] spmStart polling started: taskId = 5a62e718-7775-4eb5-9a32-eb7de347cb52
>2012-12-17 09:51:26,408 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] spmStart polling ended: taskId = 5a62e718-7775-4eb5-9a32-eb7de347cb52 task status = finished
>2012-12-17 09:51:26,426 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] spmStart polling ended, spm status: SPM
>2012-12-17 09:51:26,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5a62e718-7775-4eb5-9a32-eb7de347cb52), log id: 405dad8c
>2012-12-17 09:51:26,447 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] FINISH, HSMClearTaskVDSCommand, log id: 405dad8c
>2012-12-17 09:51:26,447 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@462ce357, log id: 5265faaf
>2012-12-17 09:51:26,735 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:51:27,429 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [4ea30a59] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:51:27,430 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7f241436
>2012-12-17 09:51:27,511 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:51:27,958 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-26) [405e01] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:51:28,185 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [405e01] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 10505760
>2012-12-17 09:51:28,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [405e01] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 38fe75c3
>2012-12-17 09:51:28,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [405e01] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:28,336 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [405e01] FINISH, SpmStopVDSCommand, log id: 38fe75c3
>2012-12-17 09:51:28,416 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [405e01] FINISH, ResetIrsVDSCommand, log id: 10505760
>2012-12-17 09:51:28,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [405e01] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 418de0b7
>2012-12-17 09:51:28,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [405e01] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:28,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [405e01] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 418de0b7
>2012-12-17 09:51:28,622 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:51:28,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4981
>2012-12-17 09:51:28,673 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4981, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 35ac9919
>2012-12-17 09:51:28,695 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] spmStart polling started: taskId = 25063552-b177-43fc-a41a-aa8e973fc67f
>2012-12-17 09:51:29,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] spmStart polling ended: taskId = 25063552-b177-43fc-a41a-aa8e973fc67f task status = finished
>2012-12-17 09:51:29,735 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] spmStart polling ended, spm status: SPM
>2012-12-17 09:51:29,736 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=25063552-b177-43fc-a41a-aa8e973fc67f), log id: 19a1fed5
>2012-12-17 09:51:29,755 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, HSMClearTaskVDSCommand, log id: 19a1fed5
>2012-12-17 09:51:29,755 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5a615f18, log id: 35ac9919
>2012-12-17 09:51:29,774 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:51:29,825 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:29,828 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 195b06b6
>2012-12-17 09:51:29,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 195b06b6
>2012-12-17 09:51:29,851 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7f241436
>2012-12-17 09:51:29,851 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [4ea30a59] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:51:29,851 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2715a3bb
>2012-12-17 09:51:29,851 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:29,853 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4945fbba
>2012-12-17 09:51:29,871 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4945fbba
>2012-12-17 09:51:29,871 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2715a3bb
>2012-12-17 09:51:29,871 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [4ea30a59] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:51:33,991 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:51:38,751 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1eb0b032] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:51:39,233 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [854c8c6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:51:39,352 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6e0020c1
>2012-12-17 09:51:39,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 499dbf8
>2012-12-17 09:51:39,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:39,522 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] FINISH, SpmStopVDSCommand, log id: 499dbf8
>2012-12-17 09:51:39,717 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] FINISH, ResetIrsVDSCommand, log id: 6e0020c1
>2012-12-17 09:51:39,728 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6def32c8
>2012-12-17 09:51:39,728 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [854c8c6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:39,728 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6def32c8
>2012-12-17 09:51:44,373 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [2c358b5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:51:50,162 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:51:50,195 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4982
>2012-12-17 09:51:50,196 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4982, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7ed879bd
>2012-12-17 09:51:50,220 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] spmStart polling started: taskId = 5a0019bc-b4d8-41aa-91db-ed9e752f3093
>2012-12-17 09:51:51,241 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] spmStart polling ended: taskId = 5a0019bc-b4d8-41aa-91db-ed9e752f3093 task status = finished
>2012-12-17 09:51:51,258 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] spmStart polling ended, spm status: SPM
>2012-12-17 09:51:51,259 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5a0019bc-b4d8-41aa-91db-ed9e752f3093), log id: 426300be
>2012-12-17 09:51:51,278 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, HSMClearTaskVDSCommand, log id: 426300be
>2012-12-17 09:51:51,278 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@71c5dbfe, log id: 7ed879bd
>2012-12-17 09:51:51,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:51:51,758 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [477daf1e] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:51:51,758 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4cfe4f7d
>2012-12-17 09:51:51,810 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:51:52,140 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [4f15776f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:51:52,322 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [4f15776f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: b98d819
>2012-12-17 09:51:52,323 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [4f15776f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 68e2bca
>2012-12-17 09:51:52,344 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [4f15776f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:52,493 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [4f15776f] FINISH, SpmStopVDSCommand, log id: 68e2bca
>2012-12-17 09:51:52,520 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [4f15776f] FINISH, ResetIrsVDSCommand, log id: b98d819
>2012-12-17 09:51:52,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [4f15776f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c3ae027
>2012-12-17 09:51:52,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [4f15776f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:52,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [4f15776f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c3ae027
>2012-12-17 09:51:52,680 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [477daf1e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:51:52,691 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [477daf1e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4983
>2012-12-17 09:51:52,692 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4983, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 24b61521
>2012-12-17 09:51:52,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] spmStart polling started: taskId = 5c0adc66-acaa-4264-a331-763d78e2af57
>2012-12-17 09:51:53,739 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] spmStart polling ended: taskId = 5c0adc66-acaa-4264-a331-763d78e2af57 task status = finished
>2012-12-17 09:51:53,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] spmStart polling ended, spm status: SPM
>2012-12-17 09:51:53,758 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5c0adc66-acaa-4264-a331-763d78e2af57), log id: 4a992b09
>2012-12-17 09:51:53,776 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, HSMClearTaskVDSCommand, log id: 4a992b09
>2012-12-17 09:51:53,776 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@dd88222, log id: 24b61521
>2012-12-17 09:51:53,842 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [477daf1e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:51:53,927 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:53,929 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1defc929
>2012-12-17 09:51:53,948 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1defc929
>2012-12-17 09:51:53,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4cfe4f7d
>2012-12-17 09:51:53,948 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [477daf1e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:51:53,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 697fb219
>2012-12-17 09:51:53,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:51:53,949 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 291c02cc
>2012-12-17 09:51:53,967 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 291c02cc
>2012-12-17 09:51:53,967 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 697fb219
>2012-12-17 09:51:53,967 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [477daf1e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:51:54,802 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:52:02,728 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [78e614fb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:52:03,197 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [53c2bb48] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:52:03,366 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [53c2bb48] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4eed7a97
>2012-12-17 09:52:03,367 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [53c2bb48] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 58be88d2
>2012-12-17 09:52:03,388 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [53c2bb48] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:52:03,608 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [53c2bb48] FINISH, SpmStopVDSCommand, log id: 58be88d2
>2012-12-17 09:52:03,695 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [53c2bb48] FINISH, ResetIrsVDSCommand, log id: 4eed7a97
>2012-12-17 09:52:03,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [53c2bb48] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40d3e26
>2012-12-17 09:52:03,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [53c2bb48] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:52:03,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [53c2bb48] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40d3e26
>2012-12-17 09:52:05,117 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [23663da4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:52:13,842 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [9d8cf45] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:52:13,896 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [9d8cf45] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4984
>2012-12-17 09:52:13,947 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4984, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67388b14
>2012-12-17 09:52:13,970 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] spmStart polling started: taskId = e7b898fa-052d-40d0-9c16-46ba5095fc5c
>2012-12-17 09:52:14,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] spmStart polling ended: taskId = e7b898fa-052d-40d0-9c16-46ba5095fc5c task status = finished
>2012-12-17 09:52:15,010 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] spmStart polling ended, spm status: SPM
>2012-12-17 09:52:15,011 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e7b898fa-052d-40d0-9c16-46ba5095fc5c), log id: 2851bf87
>2012-12-17 09:52:15,029 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] FINISH, HSMClearTaskVDSCommand, log id: 2851bf87
>2012-12-17 09:52:15,030 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@ef54cbf, log id: 67388b14
>2012-12-17 09:52:15,078 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [9d8cf45] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:52:15,283 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:52:15,283 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 34d5b0f0
>2012-12-17 09:52:15,360 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [9d8cf45] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:52:16,906 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-79) [37d03d47] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:52:17,388 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5bb36d4f
>2012-12-17 09:52:17,390 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3730d4c7
>2012-12-17 09:52:17,410 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:52:17,559 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, SpmStopVDSCommand, log id: 3730d4c7
>2012-12-17 09:52:17,607 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, ResetIrsVDSCommand, log id: 5bb36d4f
>2012-12-17 09:52:17,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6cec7f0a
>2012-12-17 09:52:17,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [37d03d47] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:52:17,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6cec7f0a
>2012-12-17 09:52:17,708 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:52:17,814 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4985
>2012-12-17 09:52:17,816 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4985, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5b26b2e1
>2012-12-17 09:52:17,839 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling started: taskId = 5f5cad79-7cb0-4fde-88b7-9209f448e794
>2012-12-17 09:52:18,862 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended: taskId = 5f5cad79-7cb0-4fde-88b7-9209f448e794 task status = finished
>2012-12-17 09:52:18,881 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended, spm status: SPM
>2012-12-17 09:52:18,882 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5f5cad79-7cb0-4fde-88b7-9209f448e794), log id: 34be4ba4
>2012-12-17 09:52:18,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMClearTaskVDSCommand, log id: 34be4ba4
>2012-12-17 09:52:18,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@39581709, log id: 5b26b2e1
>2012-12-17 09:52:18,981 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:52:19,068 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:52:19,069 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 277eb0bf
>2012-12-17 09:52:19,090 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 277eb0bf
>2012-12-17 09:52:19,090 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 34d5b0f0
>2012-12-17 09:52:19,090 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:52:19,090 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e85dfda
>2012-12-17 09:52:19,090 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:52:19,091 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 885ba77
>2012-12-17 09:52:19,110 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 885ba77
>2012-12-17 09:52:19,110 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e85dfda
>2012-12-17 09:52:19,110 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:52:27,168 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [469720a2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:52:27,801 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [7fd67d3e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:52:28,122 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [5b398282] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:52:28,372 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [5b398282] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6e6b7435
>2012-12-17 09:52:28,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [5b398282] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 62535698
>2012-12-17 09:52:28,394 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [5b398282] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:52:28,552 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [5b398282] FINISH, SpmStopVDSCommand, log id: 62535698
>2012-12-17 09:52:28,597 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [5b398282] FINISH, ResetIrsVDSCommand, log id: 6e6b7435
>2012-12-17 09:52:28,609 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [5b398282] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 69d5eb71
>2012-12-17 09:52:28,609 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5b398282] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:52:28,609 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [5b398282] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 69d5eb71
>2012-12-17 09:52:37,475 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:52:38,808 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2111ab39] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:52:38,866 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2111ab39] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4986
>2012-12-17 09:52:38,867 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4986, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6bc61824
>2012-12-17 09:52:38,891 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] spmStart polling started: taskId = d74845c2-2b1b-46d9-8ea1-3eb6382650e6
>2012-12-17 09:52:39,913 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] spmStart polling ended: taskId = d74845c2-2b1b-46d9-8ea1-3eb6382650e6 task status = finished
>2012-12-17 09:52:39,932 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] spmStart polling ended, spm status: SPM
>2012-12-17 09:52:39,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d74845c2-2b1b-46d9-8ea1-3eb6382650e6), log id: 68270543
>2012-12-17 09:52:39,952 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] FINISH, HSMClearTaskVDSCommand, log id: 68270543
>2012-12-17 09:52:39,952 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@59caa720, log id: 6bc61824
>2012-12-17 09:52:39,965 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2111ab39] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:52:40,061 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [3499e3ae] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:52:40,061 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4dfdbd3d
>2012-12-17 09:52:40,135 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2111ab39] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:52:40,525 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [2b017a7d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:52:41,718 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5f40a586
>2012-12-17 09:52:41,720 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1574aabc
>2012-12-17 09:52:41,741 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:52:42,255 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, SpmStopVDSCommand, log id: 1574aabc
>2012-12-17 09:52:42,300 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, ResetIrsVDSCommand, log id: 5f40a586
>2012-12-17 09:52:42,311 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7ec9d908
>2012-12-17 09:52:42,311 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:52:42,311 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7ec9d908
>2012-12-17 09:52:42,491 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3499e3ae] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:52:42,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3499e3ae] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4987
>2012-12-17 09:52:42,582 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4987, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7f1c2781
>2012-12-17 09:52:42,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] spmStart polling started: taskId = e3477ac1-c1b7-4483-b115-c0bb4a42aca4
>2012-12-17 09:52:43,626 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] spmStart polling ended: taskId = e3477ac1-c1b7-4483-b115-c0bb4a42aca4 task status = finished
>2012-12-17 09:52:43,644 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] spmStart polling ended, spm status: SPM
>2012-12-17 09:52:43,645 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e3477ac1-c1b7-4483-b115-c0bb4a42aca4), log id: 33138b41
>2012-12-17 09:52:43,663 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, HSMClearTaskVDSCommand, log id: 33138b41
>2012-12-17 09:52:43,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@481b7299, log id: 7f1c2781
>2012-12-17 09:52:43,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3499e3ae] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:52:43,748 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:52:43,750 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 24578e7f
>2012-12-17 09:52:43,770 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 24578e7f
>2012-12-17 09:52:43,770 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4dfdbd3d
>2012-12-17 09:52:43,770 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [3499e3ae] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:52:43,771 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 704c467c
>2012-12-17 09:52:43,771 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:52:43,772 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7f1b5bcf
>2012-12-17 09:52:43,791 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7f1b5bcf
>2012-12-17 09:52:43,791 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 704c467c
>2012-12-17 09:52:43,791 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [3499e3ae] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:52:47,751 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:52:52,578 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [53c2bb48] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:52:53,021 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [54d6e7f6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:52:53,193 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [54d6e7f6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 447f2569
>2012-12-17 09:52:53,195 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [54d6e7f6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 34a61f40
>2012-12-17 09:52:53,215 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [54d6e7f6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:52:53,383 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [54d6e7f6] FINISH, SpmStopVDSCommand, log id: 34a61f40
>2012-12-17 09:52:53,439 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [54d6e7f6] FINISH, ResetIrsVDSCommand, log id: 447f2569
>2012-12-17 09:52:53,451 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [54d6e7f6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2035f8ff
>2012-12-17 09:52:53,451 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [54d6e7f6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:52:53,451 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [54d6e7f6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2035f8ff
>2012-12-17 09:52:58,395 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:53:03,631 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5e506f3b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:53:03,684 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5e506f3b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4988
>2012-12-17 09:53:03,685 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4988, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1e8843a8
>2012-12-17 09:53:03,707 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] spmStart polling started: taskId = d0ef27b0-b4a8-4bbe-804b-7c5452423ea5
>2012-12-17 09:53:04,734 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] spmStart polling ended: taskId = d0ef27b0-b4a8-4bbe-804b-7c5452423ea5 task status = finished
>2012-12-17 09:53:04,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] spmStart polling ended, spm status: SPM
>2012-12-17 09:53:04,767 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d0ef27b0-b4a8-4bbe-804b-7c5452423ea5), log id: 2978a13c
>2012-12-17 09:53:04,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] FINISH, HSMClearTaskVDSCommand, log id: 2978a13c
>2012-12-17 09:53:04,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2ea17f69, log id: 1e8843a8
>2012-12-17 09:53:04,933 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5e506f3b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:53:05,027 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [2353b939] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:53:05,027 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 34351a
>2012-12-17 09:53:05,088 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5e506f3b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:53:05,455 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [66369950] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:53:05,609 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 50915a43
>2012-12-17 09:53:05,611 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3ba1ae9c
>2012-12-17 09:53:05,632 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [66369950] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:05,750 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, SpmStopVDSCommand, log id: 3ba1ae9c
>2012-12-17 09:53:06,926 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, ResetIrsVDSCommand, log id: 50915a43
>2012-12-17 09:53:06,937 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 171eb9a2
>2012-12-17 09:53:06,937 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [66369950] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:06,937 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 171eb9a2
>2012-12-17 09:53:07,921 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [2353b939] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:53:08,083 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [2353b939] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4989
>2012-12-17 09:53:08,085 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4989, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2f2d71aa
>2012-12-17 09:53:08,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [2353b939] spmStart polling started: taskId = f9a714c9-7bd7-4279-8065-9104d521a7d7
>2012-12-17 09:53:09,129 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [2353b939] spmStart polling ended: taskId = f9a714c9-7bd7-4279-8065-9104d521a7d7 task status = finished
>2012-12-17 09:53:09,147 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [2353b939] spmStart polling ended, spm status: SPM
>2012-12-17 09:53:09,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f9a714c9-7bd7-4279-8065-9104d521a7d7), log id: 4afa653f
>2012-12-17 09:53:09,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, HSMClearTaskVDSCommand, log id: 4afa653f
>2012-12-17 09:53:09,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7d0d318d, log id: 2f2d71aa
>2012-12-17 09:53:09,229 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [2353b939] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:53:09,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:09,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 35fc7a7d
>2012-12-17 09:53:09,441 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 35fc7a7d
>2012-12-17 09:53:09,441 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 34351a
>2012-12-17 09:53:09,442 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [2353b939] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:53:09,442 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2547293c
>2012-12-17 09:53:09,442 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:09,443 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4846def2
>2012-12-17 09:53:09,462 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4846def2
>2012-12-17 09:53:09,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2547293c
>2012-12-17 09:53:09,462 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [2353b939] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:53:18,046 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [681f9614] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:53:18,470 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [3c62925a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:53:18,772 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: ab9ce52
>2012-12-17 09:53:18,774 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6b203c3c
>2012-12-17 09:53:18,795 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:18,957 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] FINISH, SpmStopVDSCommand, log id: 6b203c3c
>2012-12-17 09:53:18,980 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] FINISH, ResetIrsVDSCommand, log id: ab9ce52
>2012-12-17 09:53:18,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 29e46064
>2012-12-17 09:53:18,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [3c62925a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:18,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 29e46064
>2012-12-17 09:53:19,432 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5b398282] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:53:29,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:53:29,201 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4990
>2012-12-17 09:53:29,203 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4990, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 79c9d4a2
>2012-12-17 09:53:29,225 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] spmStart polling started: taskId = f52a7acc-63b5-40b4-8d9e-fd01a6c452b7
>2012-12-17 09:53:30,246 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] spmStart polling ended: taskId = f52a7acc-63b5-40b4-8d9e-fd01a6c452b7 task status = finished
>2012-12-17 09:53:30,265 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] spmStart polling ended, spm status: SPM
>2012-12-17 09:53:30,266 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f52a7acc-63b5-40b4-8d9e-fd01a6c452b7), log id: 5dea7eb9
>2012-12-17 09:53:30,285 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, HSMClearTaskVDSCommand, log id: 5dea7eb9
>2012-12-17 09:53:30,285 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4e25d3da, log id: 79c9d4a2
>2012-12-17 09:53:30,323 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:53:30,475 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [2b017a7d] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:53:30,475 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 29fff22e
>2012-12-17 09:53:30,524 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:53:31,996 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [1ecb2997] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:53:32,386 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [1ecb2997] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 79c7dbf9
>2012-12-17 09:53:32,388 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [1ecb2997] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 53f6222b
>2012-12-17 09:53:32,407 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [1ecb2997] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:32,532 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [1ecb2997] FINISH, SpmStopVDSCommand, log id: 53f6222b
>2012-12-17 09:53:32,552 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [1ecb2997] FINISH, ResetIrsVDSCommand, log id: 79c7dbf9
>2012-12-17 09:53:32,562 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [1ecb2997] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2ead5ad6
>2012-12-17 09:53:32,563 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1ecb2997] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:32,563 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [1ecb2997] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2ead5ad6
>2012-12-17 09:53:32,625 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:53:32,638 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4991
>2012-12-17 09:53:32,639 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4991, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3bf58e07
>2012-12-17 09:53:32,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] spmStart polling started: taskId = 042d28b5-70a2-4749-a93a-781366de8a52
>2012-12-17 09:53:33,686 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] spmStart polling ended: taskId = 042d28b5-70a2-4749-a93a-781366de8a52 task status = finished
>2012-12-17 09:53:33,705 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] spmStart polling ended, spm status: SPM
>2012-12-17 09:53:33,706 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=042d28b5-70a2-4749-a93a-781366de8a52), log id: 44e898c0
>2012-12-17 09:53:33,725 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, HSMClearTaskVDSCommand, log id: 44e898c0
>2012-12-17 09:53:33,725 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2848898, log id: 3bf58e07
>2012-12-17 09:53:33,738 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:53:33,796 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:33,798 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2528e505
>2012-12-17 09:53:33,818 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2528e505
>2012-12-17 09:53:33,818 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 29fff22e
>2012-12-17 09:53:33,818 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [2b017a7d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:53:33,818 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 67409d1
>2012-12-17 09:53:33,818 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:33,819 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 73535fe5
>2012-12-17 09:53:33,838 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 73535fe5
>2012-12-17 09:53:33,838 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 67409d1
>2012-12-17 09:53:33,838 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [2b017a7d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:53:40,426 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [5f4cecb9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:53:42,689 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [54d6e7f6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:53:42,987 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [5bc6d47] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:53:43,130 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [5bc6d47] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7a20f1b9
>2012-12-17 09:53:43,132 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [5bc6d47] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 78a821b2
>2012-12-17 09:53:43,152 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [5bc6d47] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:43,301 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [5bc6d47] FINISH, SpmStopVDSCommand, log id: 78a821b2
>2012-12-17 09:53:43,372 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [5bc6d47] FINISH, ResetIrsVDSCommand, log id: 7a20f1b9
>2012-12-17 09:53:43,383 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [5bc6d47] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 476b2022
>2012-12-17 09:53:43,383 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [5bc6d47] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:43,383 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [5bc6d47] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 476b2022
>2012-12-17 09:53:51,493 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:53:53,479 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:53:53,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4992
>2012-12-17 09:53:53,548 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4992, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 771738d9
>2012-12-17 09:53:53,570 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] spmStart polling started: taskId = 93d6eb0f-e76f-4f0c-970f-81e1a6ae4981
>2012-12-17 09:53:54,593 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] spmStart polling ended: taskId = 93d6eb0f-e76f-4f0c-970f-81e1a6ae4981 task status = finished
>2012-12-17 09:53:54,613 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] spmStart polling ended, spm status: SPM
>2012-12-17 09:53:54,614 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=93d6eb0f-e76f-4f0c-970f-81e1a6ae4981), log id: 2b645912
>2012-12-17 09:53:54,633 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, HSMClearTaskVDSCommand, log id: 2b645912
>2012-12-17 09:53:54,633 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4913c2ff, log id: 771738d9
>2012-12-17 09:53:54,691 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:53:54,838 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [37d03d47] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:53:54,838 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6f91b12e
>2012-12-17 09:53:54,899 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:53:55,382 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-97) [37e97c0a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:53:55,508 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1bdbd6e8
>2012-12-17 09:53:55,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 684b0a2e
>2012-12-17 09:53:55,530 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:55,667 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] FINISH, SpmStopVDSCommand, log id: 684b0a2e
>2012-12-17 09:53:55,703 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] FINISH, ResetIrsVDSCommand, log id: 1bdbd6e8
>2012-12-17 09:53:55,714 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e65c77e
>2012-12-17 09:53:55,715 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [37e97c0a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:55,715 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e65c77e
>2012-12-17 09:53:56,851 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [37d03d47] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:53:56,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [37d03d47] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4993
>2012-12-17 09:53:56,950 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4993, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2ed3bf3a
>2012-12-17 09:53:56,972 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] spmStart polling started: taskId = 98e723ed-198b-4282-8030-d333e5ca3700
>2012-12-17 09:53:58,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] spmStart polling ended: taskId = 98e723ed-198b-4282-8030-d333e5ca3700 task status = finished
>2012-12-17 09:53:58,026 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] spmStart polling ended, spm status: SPM
>2012-12-17 09:53:58,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=98e723ed-198b-4282-8030-d333e5ca3700), log id: 5297a880
>2012-12-17 09:53:58,046 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, HSMClearTaskVDSCommand, log id: 5297a880
>2012-12-17 09:53:58,046 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@62b795dc, log id: 2ed3bf3a
>2012-12-17 09:53:58,111 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [37d03d47] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:53:58,982 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:58,990 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 78b94f3f
>2012-12-17 09:53:59,010 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 78b94f3f
>2012-12-17 09:53:59,011 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6f91b12e
>2012-12-17 09:53:59,011 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [37d03d47] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:53:59,011 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4e01ef04
>2012-12-17 09:53:59,011 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:53:59,012 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3238bcd5
>2012-12-17 09:53:59,031 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3238bcd5
>2012-12-17 09:53:59,031 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4e01ef04
>2012-12-17 09:53:59,031 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [37d03d47] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:54:02,239 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:54:07,689 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [469720a2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:54:08,128 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [384b399c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:54:08,313 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 40a4f8b6
>2012-12-17 09:54:08,315 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4718d6d1
>2012-12-17 09:54:08,335 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [384b399c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:54:08,606 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SpmStopVDSCommand, log id: 4718d6d1
>2012-12-17 09:54:08,710 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, ResetIrsVDSCommand, log id: 40a4f8b6
>2012-12-17 09:54:08,721 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 50229f69
>2012-12-17 09:54:08,721 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:54:08,721 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 50229f69
>2012-12-17 09:54:12,443 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7c8941c5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:54:18,915 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [79357c7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:54:19,403 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [79357c7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4994
>2012-12-17 09:54:19,404 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [79357c7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4994, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 15e67cd6
>2012-12-17 09:54:19,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [79357c7] spmStart polling started: taskId = 8061c79b-2442-4321-9bbb-c6e2d8c35ea2
>2012-12-17 09:54:20,448 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [79357c7] spmStart polling ended: taskId = 8061c79b-2442-4321-9bbb-c6e2d8c35ea2 task status = finished
>2012-12-17 09:54:20,466 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [79357c7] spmStart polling ended, spm status: SPM
>2012-12-17 09:54:20,468 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [79357c7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8061c79b-2442-4321-9bbb-c6e2d8c35ea2), log id: 2dc865ac
>2012-12-17 09:54:20,485 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [79357c7] FINISH, HSMClearTaskVDSCommand, log id: 2dc865ac
>2012-12-17 09:54:20,486 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [79357c7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3f8b16af, log id: 15e67cd6
>2012-12-17 09:54:23,425 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [79357c7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:54:24,776 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [4f3078ed] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:54:24,776 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2660eea7
>2012-12-17 09:54:24,777 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [79357c7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:54:27,760 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [5f725bc5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:54:27,857 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2ea5d03c
>2012-12-17 09:54:27,859 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 207c8bee
>2012-12-17 09:54:27,880 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:54:28,598 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, SpmStopVDSCommand, log id: 207c8bee
>2012-12-17 09:54:29,469 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, ResetIrsVDSCommand, log id: 2ea5d03c
>2012-12-17 09:54:29,479 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a8a85ec
>2012-12-17 09:54:29,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:54:29,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a8a85ec
>2012-12-17 09:54:30,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:54:30,283 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4995
>2012-12-17 09:54:30,284 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4995, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 79614e72
>2012-12-17 09:54:30,307 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] spmStart polling started: taskId = 2bb5623f-45b5-4701-8187-cd072c0d24a1
>2012-12-17 09:54:31,328 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] spmStart polling ended: taskId = 2bb5623f-45b5-4701-8187-cd072c0d24a1 task status = finished
>2012-12-17 09:54:31,347 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] spmStart polling ended, spm status: SPM
>2012-12-17 09:54:31,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2bb5623f-45b5-4701-8187-cd072c0d24a1), log id: 68c58fa2
>2012-12-17 09:54:31,367 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, HSMClearTaskVDSCommand, log id: 68c58fa2
>2012-12-17 09:54:31,367 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2590a6b, log id: 79614e72
>2012-12-17 09:54:32,042 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:54:32,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:54:32,464 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4a6efb25
>2012-12-17 09:54:32,484 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4a6efb25
>2012-12-17 09:54:32,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2660eea7
>2012-12-17 09:54:32,484 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [4f3078ed] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:54:32,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3c2a6050
>2012-12-17 09:54:32,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:54:32,486 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 262bdc64
>2012-12-17 09:54:32,504 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 262bdc64
>2012-12-17 09:54:32,504 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3c2a6050
>2012-12-17 09:54:32,505 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [4f3078ed] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:54:34,827 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [63eed75e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:54:40,370 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [2bcb8475] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:54:40,509 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-89) [3f5f0fad] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:54:40,631 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 408c0baa
>2012-12-17 09:54:40,633 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1b75f841
>2012-12-17 09:54:40,652 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:54:40,814 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, SpmStopVDSCommand, log id: 1b75f841
>2012-12-17 09:54:41,343 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, ResetIrsVDSCommand, log id: 408c0baa
>2012-12-17 09:54:41,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4d75b817
>2012-12-17 09:54:41,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:54:41,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4d75b817
>2012-12-17 09:54:45,603 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [95f0e17] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:54:51,619 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:54:52,032 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4996
>2012-12-17 09:54:52,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4996, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 65f71b9b
>2012-12-17 09:54:52,057 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling started: taskId = 172bd843-5cfb-49c1-957b-962c121bbbf2
>2012-12-17 09:54:53,081 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling ended: taskId = 172bd843-5cfb-49c1-957b-962c121bbbf2 task status = finished
>2012-12-17 09:54:53,101 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling ended, spm status: SPM
>2012-12-17 09:54:53,103 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=172bd843-5cfb-49c1-957b-962c121bbbf2), log id: 15ab979e
>2012-12-17 09:54:53,123 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, HSMClearTaskVDSCommand, log id: 15ab979e
>2012-12-17 09:54:53,123 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@346f112f, log id: 65f71b9b
>2012-12-17 09:54:53,221 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:54:53,275 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [17eab1eb] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:54:53,275 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5588c968
>2012-12-17 09:54:53,294 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:54:53,387 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [40e7f581] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:54:53,435 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2c7d0548
>2012-12-17 09:54:53,437 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 41992b4e
>2012-12-17 09:54:53,457 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:54:53,594 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, SpmStopVDSCommand, log id: 41992b4e
>2012-12-17 09:54:53,606 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, ResetIrsVDSCommand, log id: 2c7d0548
>2012-12-17 09:54:53,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 208685f
>2012-12-17 09:54:53,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [40e7f581] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:54:53,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 208685f
>2012-12-17 09:54:53,668 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:54:53,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4997
>2012-12-17 09:54:53,700 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4997, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 76e1b6dc
>2012-12-17 09:54:53,723 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] spmStart polling started: taskId = e9690ad4-b5f1-4dbf-af4f-7ca004f55df8
>2012-12-17 09:54:54,745 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] spmStart polling ended: taskId = e9690ad4-b5f1-4dbf-af4f-7ca004f55df8 task status = finished
>2012-12-17 09:54:54,764 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] spmStart polling ended, spm status: SPM
>2012-12-17 09:54:54,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e9690ad4-b5f1-4dbf-af4f-7ca004f55df8), log id: 49acaa46
>2012-12-17 09:54:54,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, HSMClearTaskVDSCommand, log id: 49acaa46
>2012-12-17 09:54:54,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@66deff73, log id: 76e1b6dc
>2012-12-17 09:54:54,916 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:54:54,940 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:54:54,941 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 721fbf43
>2012-12-17 09:54:54,962 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 721fbf43
>2012-12-17 09:54:54,962 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5588c968
>2012-12-17 09:54:54,963 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [17eab1eb] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:54:54,963 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 423d359c
>2012-12-17 09:54:54,963 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:54:54,964 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3fa1a9b0
>2012-12-17 09:54:54,984 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3fa1a9b0
>2012-12-17 09:54:54,984 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 423d359c
>2012-12-17 09:54:54,985 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [17eab1eb] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:54:57,442 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:55:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [384b399c] Checking autorecoverable hosts
>2012-12-17 09:55:00,133 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [384b399c] Autorecovering 0 hosts
>2012-12-17 09:55:00,133 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [384b399c] Checking autorecoverable hosts done
>2012-12-17 09:55:00,133 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [384b399c] Autorecovering storage domains is disabled, skipping
>2012-12-17 09:55:03,745 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [12d72057] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:55:04,104 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [4007c40f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:55:04,350 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: ff88fa6
>2012-12-17 09:55:04,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2de6d00f
>2012-12-17 09:55:04,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:04,517 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SpmStopVDSCommand, log id: 2de6d00f
>2012-12-17 09:55:04,595 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, ResetIrsVDSCommand, log id: ff88fa6
>2012-12-17 09:55:04,607 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 620c33d4
>2012-12-17 09:55:04,607 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:04,607 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 620c33d4
>2012-12-17 09:55:08,327 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [205b7df7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:55:14,739 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:55:14,788 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4998
>2012-12-17 09:55:14,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4998, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 66e777e7
>2012-12-17 09:55:14,816 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] spmStart polling started: taskId = 090598ad-5671-4a10-bb25-103d2aec8b03
>2012-12-17 09:55:15,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] spmStart polling ended: taskId = 090598ad-5671-4a10-bb25-103d2aec8b03 task status = finished
>2012-12-17 09:55:15,859 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] spmStart polling ended, spm status: SPM
>2012-12-17 09:55:15,861 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=090598ad-5671-4a10-bb25-103d2aec8b03), log id: 1331cc12
>2012-12-17 09:55:15,880 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, HSMClearTaskVDSCommand, log id: 1331cc12
>2012-12-17 09:55:15,880 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2a4cbb96, log id: 66e777e7
>2012-12-17 09:55:16,800 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:55:16,945 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:55:16,945 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 683d40f
>2012-12-17 09:55:16,945 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:55:17,155 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [2669bad8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:55:17,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 664493e8
>2012-12-17 09:55:17,378 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 395b979e
>2012-12-17 09:55:17,398 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:17,631 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] FINISH, SpmStopVDSCommand, log id: 395b979e
>2012-12-17 09:55:17,662 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] FINISH, ResetIrsVDSCommand, log id: 664493e8
>2012-12-17 09:55:17,674 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4cbc3fbb
>2012-12-17 09:55:17,674 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2669bad8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:17,674 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4cbc3fbb
>2012-12-17 09:55:17,817 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:55:17,885 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4999
>2012-12-17 09:55:17,887 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4999, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 42a51a13
>2012-12-17 09:55:17,910 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling started: taskId = 37b1b2c8-d97f-4cd6-9166-2ec5b5fd29b0
>2012-12-17 09:55:18,936 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling ended: taskId = 37b1b2c8-d97f-4cd6-9166-2ec5b5fd29b0 task status = finished
>2012-12-17 09:55:18,955 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling ended, spm status: SPM
>2012-12-17 09:55:18,957 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=37b1b2c8-d97f-4cd6-9166-2ec5b5fd29b0), log id: 17275545
>2012-12-17 09:55:18,977 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMClearTaskVDSCommand, log id: 17275545
>2012-12-17 09:55:18,977 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@174f30e3, log id: 42a51a13
>2012-12-17 09:55:19,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:55:19,269 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:19,271 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 20586737
>2012-12-17 09:55:19,293 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 20586737
>2012-12-17 09:55:19,293 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 683d40f
>2012-12-17 09:55:19,293 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:55:19,294 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7979e823
>2012-12-17 09:55:19,294 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:19,295 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 16565578
>2012-12-17 09:55:19,315 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 16565578
>2012-12-17 09:55:19,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7979e823
>2012-12-17 09:55:19,315 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:55:19,347 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:55:27,889 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:55:28,236 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [7fb5660a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:55:28,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [7fb5660a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2d5fdadc
>2012-12-17 09:55:28,416 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [7fb5660a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4e59b9cc
>2012-12-17 09:55:28,436 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [7fb5660a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:28,579 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [7fb5660a] FINISH, SpmStopVDSCommand, log id: 4e59b9cc
>2012-12-17 09:55:28,604 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [7fb5660a] FINISH, ResetIrsVDSCommand, log id: 2d5fdadc
>2012-12-17 09:55:28,615 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [7fb5660a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 46d38d19
>2012-12-17 09:55:28,615 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [7fb5660a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:28,615 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [7fb5660a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 46d38d19
>2012-12-17 09:55:30,524 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [20d09da7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:55:38,737 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [63fee8f1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:55:38,756 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [63fee8f1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5000
>2012-12-17 09:55:38,758 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5000, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62454aa3
>2012-12-17 09:55:38,781 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] spmStart polling started: taskId = ae1b3009-e724-4ba1-9b47-7545b0b31bc8
>2012-12-17 09:55:39,820 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] spmStart polling ended: taskId = ae1b3009-e724-4ba1-9b47-7545b0b31bc8 task status = finished
>2012-12-17 09:55:39,838 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] spmStart polling ended, spm status: SPM
>2012-12-17 09:55:39,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ae1b3009-e724-4ba1-9b47-7545b0b31bc8), log id: 6eab61de
>2012-12-17 09:55:39,858 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] FINISH, HSMClearTaskVDSCommand, log id: 6eab61de
>2012-12-17 09:55:39,858 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@35a66e95, log id: 62454aa3
>2012-12-17 09:55:39,928 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [63fee8f1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:55:39,978 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:55:39,978 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 42d67975
>2012-12-17 09:55:40,025 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [63fee8f1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:55:40,390 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [5a33a5b6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:55:40,593 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [5a33a5b6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17dbb04b
>2012-12-17 09:55:40,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [5a33a5b6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 12bcee3c
>2012-12-17 09:55:40,615 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [5a33a5b6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:41,221 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [5a33a5b6] FINISH, SpmStopVDSCommand, log id: 12bcee3c
>2012-12-17 09:55:41,759 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [5a33a5b6] FINISH, ResetIrsVDSCommand, log id: 17dbb04b
>2012-12-17 09:55:41,770 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [5a33a5b6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c0497ff
>2012-12-17 09:55:41,770 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [5a33a5b6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:41,770 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [5a33a5b6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c0497ff
>2012-12-17 09:55:42,243 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:55:42,252 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:55:42,295 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5001
>2012-12-17 09:55:42,296 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5001, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4b146a79
>2012-12-17 09:55:42,321 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling started: taskId = e91ace84-5987-4aa3-8606-efdc49d60b63
>2012-12-17 09:55:43,374 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended: taskId = e91ace84-5987-4aa3-8606-efdc49d60b63 task status = finished
>2012-12-17 09:55:43,393 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended, spm status: SPM
>2012-12-17 09:55:43,394 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e91ace84-5987-4aa3-8606-efdc49d60b63), log id: 167c1698
>2012-12-17 09:55:43,413 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMClearTaskVDSCommand, log id: 167c1698
>2012-12-17 09:55:43,413 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2caf60b0, log id: 4b146a79
>2012-12-17 09:55:43,435 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:55:43,495 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:43,498 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1a6cc3ed
>2012-12-17 09:55:43,519 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1a6cc3ed
>2012-12-17 09:55:43,519 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 42d67975
>2012-12-17 09:55:43,519 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:55:43,519 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5ab30a28
>2012-12-17 09:55:43,519 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:43,521 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 35204e4
>2012-12-17 09:55:43,539 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 35204e4
>2012-12-17 09:55:43,539 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5ab30a28
>2012-12-17 09:55:43,539 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:55:43,551 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [40e7f581] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:55:43,551 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: c5a11e8
>2012-12-17 09:55:43,551 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:43,553 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 27c1ea80
>2012-12-17 09:55:43,572 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 27c1ea80
>2012-12-17 09:55:43,572 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: c5a11e8
>2012-12-17 09:55:43,572 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [40e7f581] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:55:43,572 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2c0a8de4
>2012-12-17 09:55:43,572 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:43,574 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 320facef
>2012-12-17 09:55:43,592 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 320facef
>2012-12-17 09:55:43,592 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2c0a8de4
>2012-12-17 09:55:43,592 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [40e7f581] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:55:52,459 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [61b60bdd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:55:53,463 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [6c52cc2e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:55:54,210 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [6c52cc2e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3644bf4d
>2012-12-17 09:55:54,213 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [6c52cc2e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 46929707
>2012-12-17 09:55:54,233 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [6c52cc2e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:54,445 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [6c52cc2e] FINISH, SpmStopVDSCommand, log id: 46929707
>2012-12-17 09:55:54,479 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [6c52cc2e] FINISH, ResetIrsVDSCommand, log id: 3644bf4d
>2012-12-17 09:55:54,489 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [6c52cc2e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2cc77782
>2012-12-17 09:55:54,489 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [6c52cc2e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:55:54,489 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [6c52cc2e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2cc77782
>2012-12-17 09:56:03,027 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3499e3ae] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:56:04,719 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:56:04,732 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5002
>2012-12-17 09:56:04,734 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5002, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 64306403
>2012-12-17 09:56:04,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] spmStart polling started: taskId = 58077d2e-1274-420d-97fb-9c6b1a910c8b
>2012-12-17 09:56:05,779 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] spmStart polling ended: taskId = 58077d2e-1274-420d-97fb-9c6b1a910c8b task status = finished
>2012-12-17 09:56:05,797 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] spmStart polling ended, spm status: SPM
>2012-12-17 09:56:05,798 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=58077d2e-1274-420d-97fb-9c6b1a910c8b), log id: 92f90c8
>2012-12-17 09:56:05,817 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, HSMClearTaskVDSCommand, log id: 92f90c8
>2012-12-17 09:56:05,817 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@11ca3f2e, log id: 64306403
>2012-12-17 09:56:06,787 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:56:07,128 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [44cd9139] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:56:07,128 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 76cdbd8d
>2012-12-17 09:56:07,192 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:56:07,653 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [3ad075f5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:56:07,978 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6aaa3971
>2012-12-17 09:56:07,980 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 15e80d24
>2012-12-17 09:56:08,000 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:56:08,150 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, SpmStopVDSCommand, log id: 15e80d24
>2012-12-17 09:56:08,182 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, ResetIrsVDSCommand, log id: 6aaa3971
>2012-12-17 09:56:08,197 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2801f20
>2012-12-17 09:56:08,197 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3ad075f5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:56:08,197 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2801f20
>2012-12-17 09:56:08,245 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:56:08,275 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5003
>2012-12-17 09:56:08,277 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5003, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 503dc5a6
>2012-12-17 09:56:08,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] spmStart polling started: taskId = 8a409f6a-44d8-4049-8cf0-bdc80b7be207
>2012-12-17 09:56:09,320 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] spmStart polling ended: taskId = 8a409f6a-44d8-4049-8cf0-bdc80b7be207 task status = finished
>2012-12-17 09:56:09,339 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] spmStart polling ended, spm status: SPM
>2012-12-17 09:56:09,341 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8a409f6a-44d8-4049-8cf0-bdc80b7be207), log id: 22773145
>2012-12-17 09:56:09,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, HSMClearTaskVDSCommand, log id: 22773145
>2012-12-17 09:56:09,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3f43bab0, log id: 503dc5a6
>2012-12-17 09:56:09,419 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:56:09,475 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:56:09,477 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 24b455d3
>2012-12-17 09:56:09,496 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 24b455d3
>2012-12-17 09:56:09,496 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 76cdbd8d
>2012-12-17 09:56:09,496 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [44cd9139] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:56:09,497 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3b6ea278
>2012-12-17 09:56:09,497 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:56:09,498 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 209c800f
>2012-12-17 09:56:09,517 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 209c800f
>2012-12-17 09:56:09,517 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3b6ea278
>2012-12-17 09:56:09,517 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [44cd9139] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:56:13,326 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:56:18,399 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [7fb5660a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:56:18,686 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [3577b816] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:56:18,798 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [3577b816] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7d74c9c2
>2012-12-17 09:56:18,800 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [3577b816] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 511179b
>2012-12-17 09:56:18,820 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [3577b816] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:56:18,970 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [3577b816] FINISH, SpmStopVDSCommand, log id: 511179b
>2012-12-17 09:56:18,986 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [3577b816] FINISH, ResetIrsVDSCommand, log id: 7d74c9c2
>2012-12-17 09:56:18,996 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [3577b816] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d74fc06
>2012-12-17 09:56:18,996 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3577b816] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:56:18,996 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [3577b816] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d74fc06
>2012-12-17 09:56:23,615 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [4e8c7167] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:56:29,121 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:56:32,807 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5004
>2012-12-17 09:56:32,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5004, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 74a4c31a
>2012-12-17 09:56:32,833 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] spmStart polling started: taskId = 2d7b75f1-caf6-4dae-a63d-000418833f97
>2012-12-17 09:56:33,857 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] spmStart polling ended: taskId = 2d7b75f1-caf6-4dae-a63d-000418833f97 task status = finished
>2012-12-17 09:56:33,883 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] spmStart polling ended, spm status: SPM
>2012-12-17 09:56:33,884 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2d7b75f1-caf6-4dae-a63d-000418833f97), log id: 4eac3ff8
>2012-12-17 09:56:33,904 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, HSMClearTaskVDSCommand, log id: 4eac3ff8
>2012-12-17 09:56:33,904 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@56150e49, log id: 74a4c31a
>2012-12-17 09:56:33,918 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:56:34,019 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [312ad09c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:56:34,019 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6cb1fc7e
>2012-12-17 09:56:34,027 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:56:34,201 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [56592212] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:56:34,326 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2a7e6228
>2012-12-17 09:56:34,328 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 890ab47
>2012-12-17 09:56:34,348 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [56592212] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:56:34,813 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, SpmStopVDSCommand, log id: 890ab47
>2012-12-17 09:56:34,829 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, ResetIrsVDSCommand, log id: 2a7e6228
>2012-12-17 09:56:34,840 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 77eeb7d4
>2012-12-17 09:56:34,840 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [56592212] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:56:34,840 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 77eeb7d4
>2012-12-17 09:56:34,926 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:56:34,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5005
>2012-12-17 09:56:34,949 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5005, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7a68a922
>2012-12-17 09:56:34,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] spmStart polling started: taskId = dba95c9f-f7ce-4d7f-85cc-fcbc2e9d13d6
>2012-12-17 09:56:35,995 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] spmStart polling ended: taskId = dba95c9f-f7ce-4d7f-85cc-fcbc2e9d13d6 task status = finished
>2012-12-17 09:56:36,013 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] spmStart polling ended, spm status: SPM
>2012-12-17 09:56:36,014 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dba95c9f-f7ce-4d7f-85cc-fcbc2e9d13d6), log id: 217bd1c5
>2012-12-17 09:56:36,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, HSMClearTaskVDSCommand, log id: 217bd1c5
>2012-12-17 09:56:36,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4808edaf, log id: 7a68a922
>2012-12-17 09:56:37,153 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:56:37,322 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:56:37,324 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 445fea70
>2012-12-17 09:56:37,345 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 445fea70
>2012-12-17 09:56:37,345 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6cb1fc7e
>2012-12-17 09:56:37,345 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [312ad09c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:56:37,345 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 75188221
>2012-12-17 09:56:37,345 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:56:37,346 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1c5f658a
>2012-12-17 09:56:37,364 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1c5f658a
>2012-12-17 09:56:37,364 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 75188221
>2012-12-17 09:56:37,365 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [312ad09c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:56:44,141 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [6c52cc2e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:56:45,063 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [510758b1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:56:46,504 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-54) [3e0f1a8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:56:47,675 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [3e0f1a8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 20734fd7
>2012-12-17 09:56:47,677 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [3e0f1a8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 76386c4d
>2012-12-17 09:56:47,698 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [3e0f1a8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:56:48,261 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [3e0f1a8] FINISH, SpmStopVDSCommand, log id: 76386c4d
>2012-12-17 09:56:48,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [3e0f1a8] FINISH, ResetIrsVDSCommand, log id: 20734fd7
>2012-12-17 09:56:48,301 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [3e0f1a8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3d5fa654
>2012-12-17 09:56:48,301 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [3e0f1a8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:56:48,301 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [3e0f1a8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3d5fa654
>2012-12-17 09:56:54,638 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:56:58,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:56:59,107 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5006
>2012-12-17 09:56:59,109 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5006, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1ede79be
>2012-12-17 09:56:59,132 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] spmStart polling started: taskId = c2f41f76-d2f6-42f8-be61-5e7683a9b45f
>2012-12-17 09:57:00,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] spmStart polling ended: taskId = c2f41f76-d2f6-42f8-be61-5e7683a9b45f task status = finished
>2012-12-17 09:57:00,183 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] spmStart polling ended, spm status: SPM
>2012-12-17 09:57:00,185 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c2f41f76-d2f6-42f8-be61-5e7683a9b45f), log id: 7837eb1c
>2012-12-17 09:57:00,203 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, HSMClearTaskVDSCommand, log id: 7837eb1c
>2012-12-17 09:57:00,203 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1f2cb8ed, log id: 1ede79be
>2012-12-17 09:57:00,248 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:57:00,376 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-32) [3e0f3441] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:57:00,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 45507629
>2012-12-17 09:57:00,459 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:57:00,688 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [536ede7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:57:00,761 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3e86e124
>2012-12-17 09:57:00,763 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 79a16691
>2012-12-17 09:57:00,793 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [536ede7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:00,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, SpmStopVDSCommand, log id: 79a16691
>2012-12-17 09:57:01,877 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, ResetIrsVDSCommand, log id: 3e86e124
>2012-12-17 09:57:01,888 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7ac39947
>2012-12-17 09:57:01,888 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [536ede7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:01,888 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7ac39947
>2012-12-17 09:57:04,109 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:57:04,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5007
>2012-12-17 09:57:04,400 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5007, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7daa7153
>2012-12-17 09:57:04,425 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] spmStart polling started: taskId = 0fd174e1-2ea2-43a5-a2c0-e8f0ab1fe991
>2012-12-17 09:57:05,583 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] spmStart polling ended: taskId = 0fd174e1-2ea2-43a5-a2c0-e8f0ab1fe991 task status = finished
>2012-12-17 09:57:05,602 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] spmStart polling ended, spm status: SPM
>2012-12-17 09:57:05,603 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0fd174e1-2ea2-43a5-a2c0-e8f0ab1fe991), log id: 2c386a32
>2012-12-17 09:57:05,622 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, HSMClearTaskVDSCommand, log id: 2c386a32
>2012-12-17 09:57:05,622 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2bc40f4c, log id: 7daa7153
>2012-12-17 09:57:05,694 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:57:05,778 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:05,780 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 736e357e
>2012-12-17 09:57:05,799 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 736e357e
>2012-12-17 09:57:05,799 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 45507629
>2012-12-17 09:57:05,799 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-32) [3e0f3441] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:57:05,799 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 487a1e66
>2012-12-17 09:57:05,799 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:05,801 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: b8d33f1
>2012-12-17 09:57:05,819 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: b8d33f1
>2012-12-17 09:57:05,819 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 487a1e66
>2012-12-17 09:57:05,819 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-32) [3e0f3441] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:57:14,251 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [53e9685e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:57:15,151 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-35) [415bb057] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:57:15,474 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [405e01] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:57:15,622 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [415bb057] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 57da980e
>2012-12-17 09:57:15,625 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [415bb057] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3e25539e
>2012-12-17 09:57:15,644 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [415bb057] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:15,816 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [415bb057] FINISH, SpmStopVDSCommand, log id: 3e25539e
>2012-12-17 09:57:16,815 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [415bb057] FINISH, ResetIrsVDSCommand, log id: 57da980e
>2012-12-17 09:57:17,549 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [415bb057] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 25a8f3d4
>2012-12-17 09:57:17,549 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [415bb057] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:17,549 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [415bb057] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 25a8f3d4
>2012-12-17 09:57:27,790 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:57:27,907 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5008
>2012-12-17 09:57:27,909 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5008, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 37629d2a
>2012-12-17 09:57:27,931 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] spmStart polling started: taskId = 95ff63ea-9899-4692-87a0-69e530cfe11e
>2012-12-17 09:57:28,955 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] spmStart polling ended: taskId = 95ff63ea-9899-4692-87a0-69e530cfe11e task status = finished
>2012-12-17 09:57:28,973 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] spmStart polling ended, spm status: SPM
>2012-12-17 09:57:28,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=95ff63ea-9899-4692-87a0-69e530cfe11e), log id: 39d24fa5
>2012-12-17 09:57:28,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, HSMClearTaskVDSCommand, log id: 39d24fa5
>2012-12-17 09:57:28,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1e6af74b, log id: 37629d2a
>2012-12-17 09:57:29,051 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:57:29,179 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-51) [56fd87e7] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:57:29,179 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5a16a880
>2012-12-17 09:57:29,215 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:57:29,791 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [140420a5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:57:29,967 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 14c17f35
>2012-12-17 09:57:29,968 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4947f3c7
>2012-12-17 09:57:29,988 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [140420a5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:30,610 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, SpmStopVDSCommand, log id: 4947f3c7
>2012-12-17 09:57:30,654 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, ResetIrsVDSCommand, log id: 14c17f35
>2012-12-17 09:57:30,665 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76798dc
>2012-12-17 09:57:30,665 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [140420a5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:30,665 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76798dc
>2012-12-17 09:57:30,784 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:57:31,709 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5009
>2012-12-17 09:57:31,711 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5009, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e5737be
>2012-12-17 09:57:31,735 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] spmStart polling started: taskId = e09e49d7-9f36-4aa9-bdb8-fad6d11a0b8d
>2012-12-17 09:57:32,756 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] spmStart polling ended: taskId = e09e49d7-9f36-4aa9-bdb8-fad6d11a0b8d task status = finished
>2012-12-17 09:57:32,774 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] spmStart polling ended, spm status: SPM
>2012-12-17 09:57:32,775 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e09e49d7-9f36-4aa9-bdb8-fad6d11a0b8d), log id: 7fb90016
>2012-12-17 09:57:32,793 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, HSMClearTaskVDSCommand, log id: 7fb90016
>2012-12-17 09:57:32,793 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4ab4e5dc, log id: 3e5737be
>2012-12-17 09:57:32,838 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:57:32,888 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:32,890 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 114c67d6
>2012-12-17 09:57:32,910 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 114c67d6
>2012-12-17 09:57:32,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5a16a880
>2012-12-17 09:57:32,910 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-51) [56fd87e7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:57:32,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2f351b79
>2012-12-17 09:57:32,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:32,911 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 60ea5b8
>2012-12-17 09:57:32,929 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 60ea5b8
>2012-12-17 09:57:32,929 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2f351b79
>2012-12-17 09:57:32,929 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-51) [56fd87e7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:57:38,157 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:57:41,955 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [4f15776f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:57:42,325 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [2ce0ef25] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:57:42,429 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7beff2fe
>2012-12-17 09:57:42,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3d7942ad
>2012-12-17 09:57:42,450 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:42,634 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] FINISH, SpmStopVDSCommand, log id: 3d7942ad
>2012-12-17 09:57:42,717 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] FINISH, ResetIrsVDSCommand, log id: 7beff2fe
>2012-12-17 09:57:42,733 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1f6bba3b
>2012-12-17 09:57:42,733 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [2ce0ef25] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:42,733 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1f6bba3b
>2012-12-17 09:57:49,289 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:57:53,014 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [5f4cecb9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:57:53,125 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [5f4cecb9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5010
>2012-12-17 09:57:53,127 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5010, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 37dc1dab
>2012-12-17 09:57:53,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] spmStart polling started: taskId = e475a84d-4da0-4de9-a8fc-f4fda9a644a8
>2012-12-17 09:57:54,170 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] spmStart polling ended: taskId = e475a84d-4da0-4de9-a8fc-f4fda9a644a8 task status = finished
>2012-12-17 09:57:54,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] spmStart polling ended, spm status: SPM
>2012-12-17 09:57:54,190 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e475a84d-4da0-4de9-a8fc-f4fda9a644a8), log id: 182d66ef
>2012-12-17 09:57:54,208 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] FINISH, HSMClearTaskVDSCommand, log id: 182d66ef
>2012-12-17 09:57:54,209 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4cc0e14f, log id: 37dc1dab
>2012-12-17 09:57:54,311 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [5f4cecb9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:57:54,496 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [536ede7] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:57:54,496 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5fdbb076
>2012-12-17 09:57:54,570 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [5f4cecb9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:57:54,916 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [604cecb9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:57:54,961 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 490684b9
>2012-12-17 09:57:54,962 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 27090b0
>2012-12-17 09:57:54,982 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:55,117 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, SpmStopVDSCommand, log id: 27090b0
>2012-12-17 09:57:55,136 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, ResetIrsVDSCommand, log id: 490684b9
>2012-12-17 09:57:55,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 377d20fe
>2012-12-17 09:57:55,147 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:55,147 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 377d20fe
>2012-12-17 09:57:55,218 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [536ede7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:57:55,224 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [536ede7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5011
>2012-12-17 09:57:55,226 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5011, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6245bf90
>2012-12-17 09:57:55,249 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] spmStart polling started: taskId = 93be4e3f-0abc-4060-80f7-fba0139f4181
>2012-12-17 09:57:56,271 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] spmStart polling ended: taskId = 93be4e3f-0abc-4060-80f7-fba0139f4181 task status = finished
>2012-12-17 09:57:56,289 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] spmStart polling ended, spm status: SPM
>2012-12-17 09:57:56,291 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=93be4e3f-0abc-4060-80f7-fba0139f4181), log id: 3592bd3c
>2012-12-17 09:57:56,309 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, HSMClearTaskVDSCommand, log id: 3592bd3c
>2012-12-17 09:57:56,309 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@34c5c99e, log id: 6245bf90
>2012-12-17 09:57:57,115 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [536ede7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:57:57,250 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:57,253 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2f1ad9be
>2012-12-17 09:57:57,275 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2f1ad9be
>2012-12-17 09:57:57,275 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5fdbb076
>2012-12-17 09:57:57,275 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [536ede7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:57:57,276 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 655a6763
>2012-12-17 09:57:57,276 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:57:57,277 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 12e476f
>2012-12-17 09:57:57,295 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 12e476f
>2012-12-17 09:57:57,295 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 655a6763
>2012-12-17 09:57:57,295 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [536ede7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:57:59,770 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [20d09da7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:58:05,326 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [95f0e17] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:58:05,773 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [6a7edb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:58:07,501 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [6a7edb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 79b4d24e
>2012-12-17 09:58:07,503 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [6a7edb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 20d4cb71
>2012-12-17 09:58:07,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [6a7edb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:07,692 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [6a7edb] FINISH, SpmStopVDSCommand, log id: 20d4cb71
>2012-12-17 09:58:07,799 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [6a7edb] FINISH, ResetIrsVDSCommand, log id: 79b4d24e
>2012-12-17 09:58:07,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [6a7edb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e310a2a
>2012-12-17 09:58:07,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [6a7edb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:07,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [6a7edb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e310a2a
>2012-12-17 09:58:10,114 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:58:17,979 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:58:18,040 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5012
>2012-12-17 09:58:18,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5012, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e8c9236
>2012-12-17 09:58:18,064 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] spmStart polling started: taskId = ae5cdf62-8231-44be-8d23-de147d9e6df2
>2012-12-17 09:58:19,085 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] spmStart polling ended: taskId = ae5cdf62-8231-44be-8d23-de147d9e6df2 task status = finished
>2012-12-17 09:58:19,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] spmStart polling ended, spm status: SPM
>2012-12-17 09:58:19,106 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ae5cdf62-8231-44be-8d23-de147d9e6df2), log id: 6d2257e9
>2012-12-17 09:58:19,124 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, HSMClearTaskVDSCommand, log id: 6d2257e9
>2012-12-17 09:58:19,124 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@69b9b9f8, log id: 5e8c9236
>2012-12-17 09:58:19,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:58:19,600 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [384b399c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:58:19,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1d2150de
>2012-12-17 09:58:19,682 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:58:20,158 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-85) [535051e4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:58:20,255 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 595b8fa
>2012-12-17 09:58:20,257 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: b927457
>2012-12-17 09:58:20,280 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [535051e4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:20,423 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SpmStopVDSCommand, log id: b927457
>2012-12-17 09:58:20,476 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [140420a5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:58:20,498 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, ResetIrsVDSCommand, log id: 595b8fa
>2012-12-17 09:58:20,509 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5a10b439
>2012-12-17 09:58:20,509 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:20,509 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5a10b439
>2012-12-17 09:58:21,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:58:22,486 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5013
>2012-12-17 09:58:22,488 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5013, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 30d597ef
>2012-12-17 09:58:22,512 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling started: taskId = 5ea288a2-b1ed-42b4-8928-adbfd8f90816
>2012-12-17 09:58:23,533 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling ended: taskId = 5ea288a2-b1ed-42b4-8928-adbfd8f90816 task status = finished
>2012-12-17 09:58:23,551 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling ended, spm status: SPM
>2012-12-17 09:58:23,552 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5ea288a2-b1ed-42b4-8928-adbfd8f90816), log id: 12e9c602
>2012-12-17 09:58:23,570 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, HSMClearTaskVDSCommand, log id: 12e9c602
>2012-12-17 09:58:23,570 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31c6b701, log id: 30d597ef
>2012-12-17 09:58:23,756 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:58:23,812 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:23,813 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7751b068
>2012-12-17 09:58:23,833 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7751b068
>2012-12-17 09:58:23,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1d2150de
>2012-12-17 09:58:23,833 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [384b399c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:58:23,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1414b90b
>2012-12-17 09:58:23,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:23,835 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 646b8a5d
>2012-12-17 09:58:23,853 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 646b8a5d
>2012-12-17 09:58:23,853 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1414b90b
>2012-12-17 09:58:23,853 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [384b399c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:58:23,904 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-47) [7c8941c5] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:58:23,904 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6d89d6e1
>2012-12-17 09:58:23,904 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:23,906 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 31ed7e79
>2012-12-17 09:58:23,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31ed7e79
>2012-12-17 09:58:23,925 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6d89d6e1
>2012-12-17 09:58:23,925 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-47) [7c8941c5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:58:23,925 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5f0a538a
>2012-12-17 09:58:23,925 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:23,926 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2d402104
>2012-12-17 09:58:23,944 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2d402104
>2012-12-17 09:58:23,944 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5f0a538a
>2012-12-17 09:58:23,944 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-47) [7c8941c5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:58:32,311 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [477daf1e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:58:33,360 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [193f8ef] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:58:33,445 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [193f8ef] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 710269ea
>2012-12-17 09:58:33,447 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [193f8ef] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 44e9741
>2012-12-17 09:58:33,467 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [193f8ef] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:33,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [193f8ef] FINISH, SpmStopVDSCommand, log id: 44e9741
>2012-12-17 09:58:33,872 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [193f8ef] FINISH, ResetIrsVDSCommand, log id: 710269ea
>2012-12-17 09:58:33,883 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [193f8ef] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24b94ea1
>2012-12-17 09:58:33,883 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [193f8ef] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:33,883 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [193f8ef] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24b94ea1
>2012-12-17 09:58:42,826 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:58:44,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [537845d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:58:44,250 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [537845d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5014
>2012-12-17 09:58:44,251 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [537845d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5014, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e3007a2
>2012-12-17 09:58:44,273 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [537845d] spmStart polling started: taskId = 30af2b92-acd2-4e1a-a327-5fe32a8375d5
>2012-12-17 09:58:45,295 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [537845d] spmStart polling ended: taskId = 30af2b92-acd2-4e1a-a327-5fe32a8375d5 task status = finished
>2012-12-17 09:58:45,313 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [537845d] spmStart polling ended, spm status: SPM
>2012-12-17 09:58:45,314 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [537845d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=30af2b92-acd2-4e1a-a327-5fe32a8375d5), log id: 76e1ca5e
>2012-12-17 09:58:45,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [537845d] FINISH, HSMClearTaskVDSCommand, log id: 76e1ca5e
>2012-12-17 09:58:45,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [537845d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@61ad1e4b, log id: 5e3007a2
>2012-12-17 09:58:45,356 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [537845d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:58:45,391 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-61) [604cecb9] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:58:45,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 119ba3e1
>2012-12-17 09:58:45,411 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [537845d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:58:45,602 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-20) [56dc6659] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:58:46,898 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 14b4fcfe
>2012-12-17 09:58:46,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 752a5a8
>2012-12-17 09:58:46,931 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:47,072 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, SpmStopVDSCommand, log id: 752a5a8
>2012-12-17 09:58:47,276 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, ResetIrsVDSCommand, log id: 14b4fcfe
>2012-12-17 09:58:47,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3d6c6f75
>2012-12-17 09:58:47,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:47,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3d6c6f75
>2012-12-17 09:58:47,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:58:47,449 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5015
>2012-12-17 09:58:47,451 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5015, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 76287059
>2012-12-17 09:58:47,472 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] spmStart polling started: taskId = 430c9f1b-25d0-4e9e-a0ff-57d96393cc9e
>2012-12-17 09:58:48,502 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] spmStart polling ended: taskId = 430c9f1b-25d0-4e9e-a0ff-57d96393cc9e task status = finished
>2012-12-17 09:58:48,521 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] spmStart polling ended, spm status: SPM
>2012-12-17 09:58:48,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=430c9f1b-25d0-4e9e-a0ff-57d96393cc9e), log id: a95929a
>2012-12-17 09:58:48,549 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, HSMClearTaskVDSCommand, log id: a95929a
>2012-12-17 09:58:48,549 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@61cbc1e3, log id: 76287059
>2012-12-17 09:58:48,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:58:48,638 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:48,641 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3bebaa66
>2012-12-17 09:58:48,663 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3bebaa66
>2012-12-17 09:58:48,663 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 119ba3e1
>2012-12-17 09:58:48,663 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-61) [604cecb9] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:58:48,663 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6d002d45
>2012-12-17 09:58:48,663 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:48,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 22c0020
>2012-12-17 09:58:48,683 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 22c0020
>2012-12-17 09:58:48,683 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6d002d45
>2012-12-17 09:58:48,684 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-61) [604cecb9] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:58:53,893 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:58:57,566 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [405e01] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:58:57,958 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-26) [406c9a80] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:58:58,239 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [406c9a80] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 56c5744f
>2012-12-17 09:58:58,241 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [406c9a80] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 21f676b7
>2012-12-17 09:58:58,266 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [406c9a80] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:58,423 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [406c9a80] FINISH, SpmStopVDSCommand, log id: 21f676b7
>2012-12-17 09:58:58,509 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [406c9a80] FINISH, ResetIrsVDSCommand, log id: 56c5744f
>2012-12-17 09:58:58,525 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [406c9a80] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c6377b2
>2012-12-17 09:58:58,525 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [406c9a80] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:58:58,525 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [406c9a80] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c6377b2
>2012-12-17 09:59:04,463 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [511011e7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:59:08,768 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2b52624] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:59:08,799 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2b52624] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5016
>2012-12-17 09:59:08,800 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [2b52624] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5016, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 22e644e8
>2012-12-17 09:59:08,824 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [2b52624] spmStart polling started: taskId = d73944a7-d20d-43ad-9fbc-0b598b4b06f6
>2012-12-17 09:59:09,848 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [2b52624] spmStart polling ended: taskId = d73944a7-d20d-43ad-9fbc-0b598b4b06f6 task status = finished
>2012-12-17 09:59:09,866 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [2b52624] spmStart polling ended, spm status: SPM
>2012-12-17 09:59:09,867 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [2b52624] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d73944a7-d20d-43ad-9fbc-0b598b4b06f6), log id: 49fbbc3d
>2012-12-17 09:59:09,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [2b52624] FINISH, HSMClearTaskVDSCommand, log id: 49fbbc3d
>2012-12-17 09:59:09,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [2b52624] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2e036ebd, log id: 22e644e8
>2012-12-17 09:59:10,003 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2b52624] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:59:10,112 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:59:10,112 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4dbfc446
>2012-12-17 09:59:10,218 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2b52624] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:59:10,512 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [8c9ee2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:59:11,799 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 8920f4e
>2012-12-17 09:59:11,801 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 60a765cc
>2012-12-17 09:59:11,821 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:59:11,955 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] FINISH, SpmStopVDSCommand, log id: 60a765cc
>2012-12-17 09:59:12,240 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] FINISH, ResetIrsVDSCommand, log id: 8920f4e
>2012-12-17 09:59:12,251 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7edca4bb
>2012-12-17 09:59:12,251 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [8c9ee2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:59:12,251 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7edca4bb
>2012-12-17 09:59:12,352 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:59:12,397 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5017
>2012-12-17 09:59:12,399 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5017, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5620168f
>2012-12-17 09:59:12,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling started: taskId = 10869372-5f03-4651-b0bd-15111239b91c
>2012-12-17 09:59:13,450 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling ended: taskId = 10869372-5f03-4651-b0bd-15111239b91c task status = finished
>2012-12-17 09:59:13,468 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling ended, spm status: SPM
>2012-12-17 09:59:13,470 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=10869372-5f03-4651-b0bd-15111239b91c), log id: 56447a8e
>2012-12-17 09:59:13,488 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMClearTaskVDSCommand, log id: 56447a8e
>2012-12-17 09:59:13,488 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@29ca2ce1, log id: 5620168f
>2012-12-17 09:59:13,515 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:59:13,606 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:59:13,608 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 211aace3
>2012-12-17 09:59:13,626 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 211aace3
>2012-12-17 09:59:13,626 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4dbfc446
>2012-12-17 09:59:13,626 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:59:13,627 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5e4c7c0f
>2012-12-17 09:59:13,627 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:59:13,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6e65799e
>2012-12-17 09:59:13,646 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6e65799e
>2012-12-17 09:59:13,646 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5e4c7c0f
>2012-12-17 09:59:13,646 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:59:15,030 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [6c52cc2e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:59:22,489 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3499e3ae] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:59:23,283 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [7d7168a8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:59:23,452 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [7d7168a8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1287df40
>2012-12-17 09:59:23,590 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [7d7168a8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6dd1a0e0
>2012-12-17 09:59:23,610 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [7d7168a8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:59:24,239 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [7d7168a8] FINISH, SpmStopVDSCommand, log id: 6dd1a0e0
>2012-12-17 09:59:24,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [7d7168a8] FINISH, ResetIrsVDSCommand, log id: 1287df40
>2012-12-17 09:59:24,292 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [7d7168a8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4155e423
>2012-12-17 09:59:24,292 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [7d7168a8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:59:24,292 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [7d7168a8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4155e423
>2012-12-17 09:59:27,301 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:59:34,494 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [63eed75e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:59:34,572 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [63eed75e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5018
>2012-12-17 09:59:34,573 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5018, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: e03937d
>2012-12-17 09:59:34,596 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] spmStart polling started: taskId = 1829fd44-17e2-4583-be77-716d4e79443a
>2012-12-17 09:59:35,617 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] spmStart polling ended: taskId = 1829fd44-17e2-4583-be77-716d4e79443a task status = finished
>2012-12-17 09:59:35,635 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] spmStart polling ended, spm status: SPM
>2012-12-17 09:59:35,636 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1829fd44-17e2-4583-be77-716d4e79443a), log id: 95e79e6
>2012-12-17 09:59:35,655 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] FINISH, HSMClearTaskVDSCommand, log id: 95e79e6
>2012-12-17 09:59:35,655 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1994efd8, log id: e03937d
>2012-12-17 09:59:35,743 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [63eed75e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:59:36,797 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [56dc6659] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:59:36,798 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 32e7e680
>2012-12-17 09:59:37,057 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [63eed75e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:59:37,419 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [6827474] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:59:37,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 36b39bf6
>2012-12-17 09:59:37,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3d031a1c
>2012-12-17 09:59:37,867 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 09:59:37,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [6827474] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:59:38,548 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, SpmStopVDSCommand, log id: 3d031a1c
>2012-12-17 09:59:38,603 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, ResetIrsVDSCommand, log id: 36b39bf6
>2012-12-17 09:59:38,614 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 17069c68
>2012-12-17 09:59:38,614 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [6827474] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:59:38,614 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 17069c68
>2012-12-17 09:59:38,675 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 09:59:38,687 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5019
>2012-12-17 09:59:38,688 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5019, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 58d3dc08
>2012-12-17 09:59:38,710 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] spmStart polling started: taskId = 9c22f530-d793-4b5b-9638-470729d49d8a
>2012-12-17 09:59:39,733 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] spmStart polling ended: taskId = 9c22f530-d793-4b5b-9638-470729d49d8a task status = finished
>2012-12-17 09:59:39,756 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] spmStart polling ended, spm status: SPM
>2012-12-17 09:59:39,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9c22f530-d793-4b5b-9638-470729d49d8a), log id: 42b5cc43
>2012-12-17 09:59:39,776 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, HSMClearTaskVDSCommand, log id: 42b5cc43
>2012-12-17 09:59:39,776 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2d7564ec, log id: 58d3dc08
>2012-12-17 09:59:39,919 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 09:59:39,987 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:59:39,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 29975f3d
>2012-12-17 09:59:40,013 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 29975f3d
>2012-12-17 09:59:40,013 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 32e7e680
>2012-12-17 09:59:40,013 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [56dc6659] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:59:40,014 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 58c88a25
>2012-12-17 09:59:40,014 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:59:40,015 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7e9e0e46
>2012-12-17 09:59:40,033 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7e9e0e46
>2012-12-17 09:59:40,033 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 58c88a25
>2012-12-17 09:59:40,033 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [56dc6659] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:59:40,046 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [20d09da7] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 09:59:40,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6132f83c
>2012-12-17 09:59:40,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:59:40,047 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4de561b3
>2012-12-17 09:59:40,066 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4de561b3
>2012-12-17 09:59:40,066 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6132f83c
>2012-12-17 09:59:40,066 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [20d09da7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:59:40,066 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: e7156b9
>2012-12-17 09:59:40,066 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:59:40,067 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 13e4db80
>2012-12-17 09:59:40,085 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 13e4db80
>2012-12-17 09:59:40,086 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: e7156b9
>2012-12-17 09:59:40,086 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [20d09da7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 09:59:48,836 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [37d03d47] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 09:59:49,372 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-79) [470bd088] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 09:59:49,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [470bd088] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 68f47436
>2012-12-17 09:59:49,708 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [470bd088] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 622fde75
>2012-12-17 09:59:49,728 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [470bd088] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:59:50,288 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [470bd088] FINISH, SpmStopVDSCommand, log id: 622fde75
>2012-12-17 09:59:50,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [470bd088] FINISH, ResetIrsVDSCommand, log id: 68f47436
>2012-12-17 09:59:50,402 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [470bd088] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7f8aa7ff
>2012-12-17 09:59:50,402 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [470bd088] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 09:59:50,402 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [470bd088] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7f8aa7ff
>2012-12-17 09:59:59,087 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:00:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Checking autorecoverable hosts
>2012-12-17 10:00:00,107 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Autorecovering 0 hosts
>2012-12-17 10:00:00,107 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Checking autorecoverable hosts done
>2012-12-17 10:00:00,107 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Autorecovering storage domains is disabled, skipping
>2012-12-17 10:00:00,708 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [8c9ee2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:00:00,755 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [8c9ee2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5020
>2012-12-17 10:00:00,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5020, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6c33d71
>2012-12-17 10:00:00,778 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] spmStart polling started: taskId = 0a344dba-9a60-4bd2-ad99-0852d7a0f996
>2012-12-17 10:00:01,801 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] spmStart polling ended: taskId = 0a344dba-9a60-4bd2-ad99-0852d7a0f996 task status = finished
>2012-12-17 10:00:01,820 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] spmStart polling ended, spm status: SPM
>2012-12-17 10:00:01,821 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0a344dba-9a60-4bd2-ad99-0852d7a0f996), log id: 4418f5fa
>2012-12-17 10:00:01,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] FINISH, HSMClearTaskVDSCommand, log id: 4418f5fa
>2012-12-17 10:00:01,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42bd1303, log id: 6c33d71
>2012-12-17 10:00:01,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [8c9ee2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:00:02,181 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:00:02,181 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6ccb5ffb
>2012-12-17 10:00:02,239 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [8c9ee2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:00:03,145 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [786860d5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:00:03,349 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3ad38373
>2012-12-17 10:00:03,351 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 48c20b0f
>2012-12-17 10:00:03,371 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [786860d5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:03,530 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, SpmStopVDSCommand, log id: 48c20b0f
>2012-12-17 10:00:03,660 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, ResetIrsVDSCommand, log id: 3ad38373
>2012-12-17 10:00:03,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1b0e5dc9
>2012-12-17 10:00:03,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [786860d5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:03,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1b0e5dc9
>2012-12-17 10:00:03,791 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:00:03,826 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5021
>2012-12-17 10:00:03,827 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5021, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 398ffdad
>2012-12-17 10:00:03,849 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling started: taskId = f3c01183-d47e-46ac-9212-acce2284526e
>2012-12-17 10:00:04,870 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling ended: taskId = f3c01183-d47e-46ac-9212-acce2284526e task status = finished
>2012-12-17 10:00:04,899 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling ended, spm status: SPM
>2012-12-17 10:00:04,901 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f3c01183-d47e-46ac-9212-acce2284526e), log id: 2893e673
>2012-12-17 10:00:04,919 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMClearTaskVDSCommand, log id: 2893e673
>2012-12-17 10:00:04,919 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5843abb4, log id: 398ffdad
>2012-12-17 10:00:05,093 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:00:05,249 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:05,254 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 253c5937
>2012-12-17 10:00:05,273 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 253c5937
>2012-12-17 10:00:05,273 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6ccb5ffb
>2012-12-17 10:00:05,273 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:00:05,274 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1d053cfa
>2012-12-17 10:00:05,274 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:05,275 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6e3b3a7f
>2012-12-17 10:00:05,294 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6e3b3a7f
>2012-12-17 10:00:05,294 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1d053cfa
>2012-12-17 10:00:05,294 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:00:09,596 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:00:13,861 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [7d7168a8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:00:14,072 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [44b78d4a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:00:14,171 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [44b78d4a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 219606f7
>2012-12-17 10:00:14,173 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [44b78d4a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 344864ee
>2012-12-17 10:00:14,192 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [44b78d4a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:14,385 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [44b78d4a] FINISH, SpmStopVDSCommand, log id: 344864ee
>2012-12-17 10:00:14,444 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [44b78d4a] FINISH, ResetIrsVDSCommand, log id: 219606f7
>2012-12-17 10:00:14,455 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [44b78d4a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1b4f1b2b
>2012-12-17 10:00:14,455 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [44b78d4a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:14,455 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [44b78d4a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1b4f1b2b
>2012-12-17 10:00:19,890 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3ad075f5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:00:24,580 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [4f008eee] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:00:24,633 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [4f008eee] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5022
>2012-12-17 10:00:24,634 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5022, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 301bfacd
>2012-12-17 10:00:24,656 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] spmStart polling started: taskId = 93be342a-9207-4c98-835e-8f19c2ed69c6
>2012-12-17 10:00:25,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] spmStart polling ended: taskId = 93be342a-9207-4c98-835e-8f19c2ed69c6 task status = finished
>2012-12-17 10:00:25,697 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] spmStart polling ended, spm status: SPM
>2012-12-17 10:00:25,698 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=93be342a-9207-4c98-835e-8f19c2ed69c6), log id: 48372c43
>2012-12-17 10:00:25,717 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] FINISH, HSMClearTaskVDSCommand, log id: 48372c43
>2012-12-17 10:00:25,717 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1e3b07bf, log id: 301bfacd
>2012-12-17 10:00:25,767 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [4f008eee] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:00:27,261 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5fb78147] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:00:27,261 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 31857b5a
>2012-12-17 10:00:27,406 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [4f008eee] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:00:28,211 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-15) [2a110a6a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:00:28,612 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [2a110a6a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4804d7f5
>2012-12-17 10:00:28,613 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [2a110a6a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f3243f8
>2012-12-17 10:00:28,633 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [2a110a6a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:28,791 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [2a110a6a] FINISH, SpmStopVDSCommand, log id: 4f3243f8
>2012-12-17 10:00:28,825 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [2a110a6a] FINISH, ResetIrsVDSCommand, log id: 4804d7f5
>2012-12-17 10:00:28,835 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [2a110a6a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 29e5a356
>2012-12-17 10:00:28,835 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [2a110a6a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:28,835 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [2a110a6a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 29e5a356
>2012-12-17 10:00:28,881 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:00:28,988 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5023
>2012-12-17 10:00:28,989 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5023, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 64bd0a5e
>2012-12-17 10:00:29,011 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling started: taskId = 665e0c31-21f9-4cc3-a366-7235eb915651
>2012-12-17 10:00:30,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling ended: taskId = 665e0c31-21f9-4cc3-a366-7235eb915651 task status = finished
>2012-12-17 10:00:30,053 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling ended, spm status: SPM
>2012-12-17 10:00:30,054 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=665e0c31-21f9-4cc3-a366-7235eb915651), log id: 47b3c6b3
>2012-12-17 10:00:30,089 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, HSMClearTaskVDSCommand, log id: 47b3c6b3
>2012-12-17 10:00:30,089 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@29bc2cdb, log id: 64bd0a5e
>2012-12-17 10:00:30,311 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:00:30,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:30,372 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 52b91cb0
>2012-12-17 10:00:30,393 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 52b91cb0
>2012-12-17 10:00:30,394 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 31857b5a
>2012-12-17 10:00:30,394 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5fb78147] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:00:30,394 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 414935d8
>2012-12-17 10:00:30,394 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:30,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4a67838c
>2012-12-17 10:00:30,414 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4a67838c
>2012-12-17 10:00:30,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 414935d8
>2012-12-17 10:00:30,414 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5fb78147] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:00:31,230 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3577b816] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:00:38,920 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [5a33a5b6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:00:39,078 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [4d407d58] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:00:39,213 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5d04f55d
>2012-12-17 10:00:39,215 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 655cf38d
>2012-12-17 10:00:39,236 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:39,625 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] FINISH, SpmStopVDSCommand, log id: 655cf38d
>2012-12-17 10:00:39,723 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] FINISH, ResetIrsVDSCommand, log id: 5d04f55d
>2012-12-17 10:00:39,814 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 769099
>2012-12-17 10:00:39,814 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4d407d58] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:39,814 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 769099
>2012-12-17 10:00:42,777 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:00:49,924 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [33bbf9d8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:00:49,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [33bbf9d8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5024
>2012-12-17 10:00:49,952 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5024, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44471110
>2012-12-17 10:00:49,975 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] spmStart polling started: taskId = eab553e4-00a3-49d6-957d-c81c8f28adf9
>2012-12-17 10:00:50,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] spmStart polling ended: taskId = eab553e4-00a3-49d6-957d-c81c8f28adf9 task status = finished
>2012-12-17 10:00:51,015 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] spmStart polling ended, spm status: SPM
>2012-12-17 10:00:51,016 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=eab553e4-00a3-49d6-957d-c81c8f28adf9), log id: 21682b71
>2012-12-17 10:00:51,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, HSMClearTaskVDSCommand, log id: 21682b71
>2012-12-17 10:00:51,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3dfc18d9, log id: 44471110
>2012-12-17 10:00:52,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [33bbf9d8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:00:52,948 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [4d8dc1c4] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:00:52,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1421d86b
>2012-12-17 10:00:53,037 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [33bbf9d8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:00:53,567 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [742a6686] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:00:53,788 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [742a6686] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 76d8fca7
>2012-12-17 10:00:53,789 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [742a6686] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6a9b6d0b
>2012-12-17 10:00:53,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [742a6686] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:53,958 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [742a6686] FINISH, SpmStopVDSCommand, log id: 6a9b6d0b
>2012-12-17 10:00:53,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [742a6686] FINISH, ResetIrsVDSCommand, log id: 76d8fca7
>2012-12-17 10:00:53,994 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [742a6686] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 234ea45
>2012-12-17 10:00:53,994 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [742a6686] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:53,994 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [742a6686] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 234ea45
>2012-12-17 10:00:54,184 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:00:54,213 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5025
>2012-12-17 10:00:54,214 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5025, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 28db5859
>2012-12-17 10:00:54,237 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] spmStart polling started: taskId = af6e4c20-ea02-4d0e-8f1f-91f3f6857e2e
>2012-12-17 10:00:55,259 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] spmStart polling ended: taskId = af6e4c20-ea02-4d0e-8f1f-91f3f6857e2e task status = finished
>2012-12-17 10:00:55,277 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] spmStart polling ended, spm status: SPM
>2012-12-17 10:00:55,278 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=af6e4c20-ea02-4d0e-8f1f-91f3f6857e2e), log id: 1befdf1c
>2012-12-17 10:00:55,297 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, HSMClearTaskVDSCommand, log id: 1befdf1c
>2012-12-17 10:00:55,297 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2120b598, log id: 28db5859
>2012-12-17 10:00:55,347 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:00:55,584 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:55,586 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 187dc720
>2012-12-17 10:00:55,608 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 187dc720
>2012-12-17 10:00:55,608 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1421d86b
>2012-12-17 10:00:55,608 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [4d8dc1c4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:00:55,608 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 699c126f
>2012-12-17 10:00:55,608 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:00:55,609 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 135ab894
>2012-12-17 10:00:55,627 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 135ab894
>2012-12-17 10:00:55,627 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 699c126f
>2012-12-17 10:00:55,628 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [4d8dc1c4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:01:03,370 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1ecb2997] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:01:04,388 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [44b78d4a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:01:04,760 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [569d76f1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:01:04,821 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6753cb6d
>2012-12-17 10:01:04,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5bdf9c31
>2012-12-17 10:01:04,842 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:01:05,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, SpmStopVDSCommand, log id: 5bdf9c31
>2012-12-17 10:01:05,016 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, ResetIrsVDSCommand, log id: 6753cb6d
>2012-12-17 10:01:05,027 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 450d893c
>2012-12-17 10:01:05,027 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [569d76f1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:01:05,027 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 450d893c
>2012-12-17 10:01:13,936 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [4bbb254] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:01:15,145 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:01:15,219 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5026
>2012-12-17 10:01:15,220 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5026, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1c479c82
>2012-12-17 10:01:15,243 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] spmStart polling started: taskId = aed7acca-67de-4d97-8e35-a018673077dc
>2012-12-17 10:01:16,264 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] spmStart polling ended: taskId = aed7acca-67de-4d97-8e35-a018673077dc task status = finished
>2012-12-17 10:01:16,283 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] spmStart polling ended, spm status: SPM
>2012-12-17 10:01:16,285 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=aed7acca-67de-4d97-8e35-a018673077dc), log id: 5b925b8c
>2012-12-17 10:01:16,303 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, HSMClearTaskVDSCommand, log id: 5b925b8c
>2012-12-17 10:01:16,303 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@65b8734b, log id: 1c479c82
>2012-12-17 10:01:17,032 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:01:17,254 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [6827474] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:01:17,254 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7223f2cc
>2012-12-17 10:01:17,271 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:01:17,423 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [53c52fa0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:01:17,468 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2335a941
>2012-12-17 10:01:17,469 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f493480
>2012-12-17 10:01:17,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:01:17,615 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] FINISH, SpmStopVDSCommand, log id: 4f493480
>2012-12-17 10:01:17,631 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] FINISH, ResetIrsVDSCommand, log id: 2335a941
>2012-12-17 10:01:17,642 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 15bd1d2f
>2012-12-17 10:01:17,642 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [53c52fa0] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:01:17,642 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 15bd1d2f
>2012-12-17 10:01:17,727 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [6827474] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:01:17,842 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [6827474] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5027
>2012-12-17 10:01:17,843 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5027, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 537babf6
>2012-12-17 10:01:17,867 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [6827474] spmStart polling started: taskId = d8a398fe-8f71-4455-8a4e-1f4b66c496ec
>2012-12-17 10:01:18,889 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [6827474] spmStart polling ended: taskId = d8a398fe-8f71-4455-8a4e-1f4b66c496ec task status = finished
>2012-12-17 10:01:18,907 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [6827474] spmStart polling ended, spm status: SPM
>2012-12-17 10:01:18,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d8a398fe-8f71-4455-8a4e-1f4b66c496ec), log id: 736b7147
>2012-12-17 10:01:18,927 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, HSMClearTaskVDSCommand, log id: 736b7147
>2012-12-17 10:01:18,927 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@12da2784, log id: 537babf6
>2012-12-17 10:01:18,960 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [6827474] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:01:18,980 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:01:18,982 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 626b1671
>2012-12-17 10:01:19,001 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 626b1671
>2012-12-17 10:01:19,001 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7223f2cc
>2012-12-17 10:01:19,001 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [6827474] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:01:19,002 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 494dd58a
>2012-12-17 10:01:19,002 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:01:19,003 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 35ef4b77
>2012-12-17 10:01:19,022 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 35ef4b77
>2012-12-17 10:01:19,022 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 494dd58a
>2012-12-17 10:01:19,022 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [6827474] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:01:24,374 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:01:27,875 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:01:28,540 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [5cec762d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:01:28,747 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [5cec762d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: da67bef
>2012-12-17 10:01:28,750 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [5cec762d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e317ef6
>2012-12-17 10:01:28,772 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [5cec762d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:01:28,934 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [5cec762d] FINISH, SpmStopVDSCommand, log id: 7e317ef6
>2012-12-17 10:01:28,994 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [5cec762d] FINISH, ResetIrsVDSCommand, log id: da67bef
>2012-12-17 10:01:29,005 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [5cec762d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 547b1cdb
>2012-12-17 10:01:29,005 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [5cec762d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:01:29,005 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [5cec762d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 547b1cdb
>2012-12-17 10:01:34,843 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [511011e7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:01:39,120 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [854c8c6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:01:39,216 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [854c8c6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5028
>2012-12-17 10:01:39,218 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5028, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6947245b
>2012-12-17 10:01:39,241 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] spmStart polling started: taskId = d8464e7c-9194-4e3c-83a3-79253d717741
>2012-12-17 10:01:40,265 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] spmStart polling ended: taskId = d8464e7c-9194-4e3c-83a3-79253d717741 task status = finished
>2012-12-17 10:01:40,283 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] spmStart polling ended, spm status: SPM
>2012-12-17 10:01:40,284 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d8464e7c-9194-4e3c-83a3-79253d717741), log id: 74e245b9
>2012-12-17 10:01:40,302 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] FINISH, HSMClearTaskVDSCommand, log id: 74e245b9
>2012-12-17 10:01:40,302 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@334fb0a2, log id: 6947245b
>2012-12-17 10:01:40,387 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [854c8c6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:01:40,475 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [140420a5] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:01:40,475 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 696aabdb
>2012-12-17 10:01:40,487 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [854c8c6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:01:40,875 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [75114a28] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:01:42,387 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [75114a28] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6613c46a
>2012-12-17 10:01:42,389 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [75114a28] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5d89812d
>2012-12-17 10:01:42,410 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [75114a28] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:01:42,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [75114a28] FINISH, SpmStopVDSCommand, log id: 5d89812d
>2012-12-17 10:01:42,970 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [75114a28] FINISH, ResetIrsVDSCommand, log id: 6613c46a
>2012-12-17 10:01:42,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [75114a28] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5c2ea578
>2012-12-17 10:01:42,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [75114a28] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:01:42,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [75114a28] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5c2ea578
>2012-12-17 10:01:43,079 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [140420a5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:01:43,103 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [140420a5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5029
>2012-12-17 10:01:43,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5029, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: d14a10b
>2012-12-17 10:01:43,126 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [140420a5] spmStart polling started: taskId = 69f0ce72-5f18-477b-9771-41c0a269bbc6
>2012-12-17 10:01:44,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [140420a5] spmStart polling ended: taskId = 69f0ce72-5f18-477b-9771-41c0a269bbc6 task status = finished
>2012-12-17 10:01:44,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [140420a5] spmStart polling ended, spm status: SPM
>2012-12-17 10:01:44,168 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=69f0ce72-5f18-477b-9771-41c0a269bbc6), log id: 77f7b1e4
>2012-12-17 10:01:44,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, HSMClearTaskVDSCommand, log id: 77f7b1e4
>2012-12-17 10:01:44,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@26c02d44, log id: d14a10b
>2012-12-17 10:01:44,195 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [140420a5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:01:44,210 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:01:44,214 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1d2fe023
>2012-12-17 10:01:44,234 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1d2fe023
>2012-12-17 10:01:44,234 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 696aabdb
>2012-12-17 10:01:44,234 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [140420a5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:01:44,234 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 63f2c472
>2012-12-17 10:01:44,235 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:01:44,236 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7981023f
>2012-12-17 10:01:44,254 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7981023f
>2012-12-17 10:01:44,255 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 63f2c472
>2012-12-17 10:01:44,255 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [140420a5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:01:45,258 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [78864454] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:01:53,114 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7712cc49] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:01:53,555 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-58) [39a6c2d3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:01:53,870 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [39a6c2d3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 69f85066
>2012-12-17 10:01:53,872 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [39a6c2d3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 600daa0a
>2012-12-17 10:01:53,892 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [39a6c2d3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:01:54,037 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [39a6c2d3] FINISH, SpmStopVDSCommand, log id: 600daa0a
>2012-12-17 10:01:54,206 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [39a6c2d3] FINISH, ResetIrsVDSCommand, log id: 69f85066
>2012-12-17 10:01:54,217 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [39a6c2d3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: e81ec2c
>2012-12-17 10:01:54,217 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [39a6c2d3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:01:54,217 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [39a6c2d3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: e81ec2c
>2012-12-17 10:01:55,572 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:02:04,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [23663da4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:02:04,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [23663da4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5030
>2012-12-17 10:02:04,511 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [23663da4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5030, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5557de58
>2012-12-17 10:02:04,533 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [23663da4] spmStart polling started: taskId = ec0535f4-e1ce-4fda-a34e-228d48f1f0f6
>2012-12-17 10:02:05,556 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [23663da4] spmStart polling ended: taskId = ec0535f4-e1ce-4fda-a34e-228d48f1f0f6 task status = finished
>2012-12-17 10:02:05,574 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [23663da4] spmStart polling ended, spm status: SPM
>2012-12-17 10:02:05,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [23663da4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ec0535f4-e1ce-4fda-a34e-228d48f1f0f6), log id: 25047d0b
>2012-12-17 10:02:05,594 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [23663da4] FINISH, HSMClearTaskVDSCommand, log id: 25047d0b
>2012-12-17 10:02:05,594 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [23663da4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@49ad7029, log id: 5557de58
>2012-12-17 10:02:05,780 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [23663da4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:02:08,287 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [3f5f0fad] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:02:08,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 47af4761
>2012-12-17 10:02:08,287 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [23663da4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:02:09,654 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [26a9278b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:02:09,879 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 24a67139
>2012-12-17 10:02:09,881 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 49cf5dbe
>2012-12-17 10:02:09,901 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:10,041 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, SpmStopVDSCommand, log id: 49cf5dbe
>2012-12-17 10:02:10,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, ResetIrsVDSCommand, log id: 24a67139
>2012-12-17 10:02:10,157 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 35f3ec9e
>2012-12-17 10:02:10,157 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [26a9278b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:10,157 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 35f3ec9e
>2012-12-17 10:02:10,341 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:02:10,361 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5031
>2012-12-17 10:02:10,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5031, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: a3b680b
>2012-12-17 10:02:10,384 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] spmStart polling started: taskId = 4a6dad52-bf6f-4474-924b-d432faeef8af
>2012-12-17 10:02:11,404 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] spmStart polling ended: taskId = 4a6dad52-bf6f-4474-924b-d432faeef8af task status = finished
>2012-12-17 10:02:11,423 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] spmStart polling ended, spm status: SPM
>2012-12-17 10:02:11,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4a6dad52-bf6f-4474-924b-d432faeef8af), log id: 12a11d42
>2012-12-17 10:02:11,442 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, HSMClearTaskVDSCommand, log id: 12a11d42
>2012-12-17 10:02:11,442 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@f87e19c, log id: a3b680b
>2012-12-17 10:02:11,792 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:02:11,993 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:11,995 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6a2bacc2
>2012-12-17 10:02:12,013 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6a2bacc2
>2012-12-17 10:02:12,013 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 47af4761
>2012-12-17 10:02:12,013 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [3f5f0fad] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:02:12,014 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 595b2b02
>2012-12-17 10:02:12,014 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:12,015 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3f7607f9
>2012-12-17 10:02:12,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3f7607f9
>2012-12-17 10:02:12,035 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 595b2b02
>2012-12-17 10:02:12,035 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [3f5f0fad] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:02:18,994 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [66369950] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:02:20,395 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [5cec762d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:02:21,841 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [3ced8e0c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:02:22,045 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [3ced8e0c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3f322719
>2012-12-17 10:02:22,048 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [3ced8e0c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 48e63073
>2012-12-17 10:02:22,068 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [3ced8e0c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:22,443 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [3ced8e0c] FINISH, SpmStopVDSCommand, log id: 48e63073
>2012-12-17 10:02:22,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [3ced8e0c] FINISH, ResetIrsVDSCommand, log id: 3f322719
>2012-12-17 10:02:22,491 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [3ced8e0c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 75379658
>2012-12-17 10:02:22,491 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [3ced8e0c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:22,491 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [3ced8e0c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 75379658
>2012-12-17 10:02:29,343 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:02:32,532 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:02:32,568 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5032
>2012-12-17 10:02:32,569 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5032, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4526c1b3
>2012-12-17 10:02:32,592 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] spmStart polling started: taskId = dc4e42b1-03b3-41cf-9ecd-27fb7dced61c
>2012-12-17 10:02:33,614 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] spmStart polling ended: taskId = dc4e42b1-03b3-41cf-9ecd-27fb7dced61c task status = finished
>2012-12-17 10:02:33,632 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] spmStart polling ended, spm status: SPM
>2012-12-17 10:02:33,634 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dc4e42b1-03b3-41cf-9ecd-27fb7dced61c), log id: 7c3b4e74
>2012-12-17 10:02:33,652 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, HSMClearTaskVDSCommand, log id: 7c3b4e74
>2012-12-17 10:02:33,652 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@199be164, log id: 4526c1b3
>2012-12-17 10:02:33,694 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:02:33,822 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:02:33,822 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1424dc66
>2012-12-17 10:02:33,875 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:02:34,335 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [5920de03] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:02:34,569 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 227b470a
>2012-12-17 10:02:34,570 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3496d411
>2012-12-17 10:02:34,591 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [5920de03] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:34,718 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, SpmStopVDSCommand, log id: 3496d411
>2012-12-17 10:02:34,749 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, ResetIrsVDSCommand, log id: 227b470a
>2012-12-17 10:02:34,760 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 636a4c7e
>2012-12-17 10:02:34,761 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [5920de03] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:34,761 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 636a4c7e
>2012-12-17 10:02:34,888 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:02:34,903 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5033
>2012-12-17 10:02:34,905 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5033, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 60d5ba0c
>2012-12-17 10:02:34,926 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling started: taskId = 21a8037b-a5d0-4ad2-a5db-6cd521c8dd0c
>2012-12-17 10:02:35,948 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling ended: taskId = 21a8037b-a5d0-4ad2-a5db-6cd521c8dd0c task status = finished
>2012-12-17 10:02:35,966 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling ended, spm status: SPM
>2012-12-17 10:02:35,967 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=21a8037b-a5d0-4ad2-a5db-6cd521c8dd0c), log id: 7130ba21
>2012-12-17 10:02:35,987 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMClearTaskVDSCommand, log id: 7130ba21
>2012-12-17 10:02:35,987 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@360c0695, log id: 60d5ba0c
>2012-12-17 10:02:36,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:02:36,872 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:36,877 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 60411f57
>2012-12-17 10:02:36,898 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 60411f57
>2012-12-17 10:02:36,898 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1424dc66
>2012-12-17 10:02:36,898 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:02:36,898 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: dcb7405
>2012-12-17 10:02:36,898 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:36,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4ee22466
>2012-12-17 10:02:36,920 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4ee22466
>2012-12-17 10:02:36,920 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: dcb7405
>2012-12-17 10:02:36,920 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:02:39,627 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:02:44,927 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [29b31be1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:02:45,030 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [4a8b829d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:02:45,106 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2bdeffe6
>2012-12-17 10:02:45,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5d20e91a
>2012-12-17 10:02:45,128 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:45,316 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] FINISH, SpmStopVDSCommand, log id: 5d20e91a
>2012-12-17 10:02:45,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] FINISH, ResetIrsVDSCommand, log id: 2bdeffe6
>2012-12-17 10:02:45,380 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: dd25436
>2012-12-17 10:02:45,380 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [4a8b829d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:45,380 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: dd25436
>2012-12-17 10:02:49,962 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3ad075f5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:02:55,529 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [53c52fa0] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:02:55,603 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [53c52fa0] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5034
>2012-12-17 10:02:55,604 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5034, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 786b6820
>2012-12-17 10:02:55,627 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] spmStart polling started: taskId = bc40b477-fe1b-4827-af75-b6c54b5963ba
>2012-12-17 10:02:56,647 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] spmStart polling ended: taskId = bc40b477-fe1b-4827-af75-b6c54b5963ba task status = finished
>2012-12-17 10:02:56,665 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] spmStart polling ended, spm status: SPM
>2012-12-17 10:02:56,666 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bc40b477-fe1b-4827-af75-b6c54b5963ba), log id: 49e4b11c
>2012-12-17 10:02:56,684 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] FINISH, HSMClearTaskVDSCommand, log id: 49e4b11c
>2012-12-17 10:02:56,684 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4527cb2a, log id: 786b6820
>2012-12-17 10:02:57,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [53c52fa0] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:02:57,452 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [56dc6659] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:02:57,453 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4c25ec6f
>2012-12-17 10:02:57,453 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [53c52fa0] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:02:57,803 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:02:57,988 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 28577e39
>2012-12-17 10:02:57,990 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 40027332
>2012-12-17 10:02:58,010 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:58,195 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] FINISH, SpmStopVDSCommand, log id: 40027332
>2012-12-17 10:02:58,230 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] FINISH, ResetIrsVDSCommand, log id: 28577e39
>2012-12-17 10:02:58,241 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e0f4c4d
>2012-12-17 10:02:58,241 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:58,241 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e0f4c4d
>2012-12-17 10:02:58,353 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:02:58,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5035
>2012-12-17 10:02:58,399 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5035, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 561f9b9f
>2012-12-17 10:02:58,423 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] spmStart polling started: taskId = 14e9429e-dc0b-40f4-a320-cf27d181752e
>2012-12-17 10:02:59,445 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] spmStart polling ended: taskId = 14e9429e-dc0b-40f4-a320-cf27d181752e task status = finished
>2012-12-17 10:02:59,463 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] spmStart polling ended, spm status: SPM
>2012-12-17 10:02:59,464 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=14e9429e-dc0b-40f4-a320-cf27d181752e), log id: 4d165fe3
>2012-12-17 10:02:59,483 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, HSMClearTaskVDSCommand, log id: 4d165fe3
>2012-12-17 10:02:59,483 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@77f1579e, log id: 561f9b9f
>2012-12-17 10:02:59,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:02:59,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:59,600 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 42ef9766
>2012-12-17 10:02:59,623 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 42ef9766
>2012-12-17 10:02:59,623 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4c25ec6f
>2012-12-17 10:02:59,623 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [56dc6659] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:02:59,623 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7fbccbde
>2012-12-17 10:02:59,623 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:02:59,624 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3c467537
>2012-12-17 10:02:59,643 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3c467537
>2012-12-17 10:02:59,643 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7fbccbde
>2012-12-17 10:02:59,643 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [56dc6659] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:03:00,276 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:03:08,422 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [16b67351] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:03:08,771 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [7dff3777] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:03:09,078 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 570ededb
>2012-12-17 10:03:09,080 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6d968357
>2012-12-17 10:03:09,100 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:03:09,385 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, SpmStopVDSCommand, log id: 6d968357
>2012-12-17 10:03:09,445 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, ResetIrsVDSCommand, log id: 570ededb
>2012-12-17 10:03:09,455 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3cdc8682
>2012-12-17 10:03:09,455 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [7dff3777] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:03:09,456 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3cdc8682
>2012-12-17 10:03:10,650 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:03:19,560 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5b398282] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:03:19,600 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5b398282] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5036
>2012-12-17 10:03:19,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5b398282] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5036, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 65ffd80d
>2012-12-17 10:03:19,624 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5b398282] spmStart polling started: taskId = 7926acf1-8885-40a0-b9df-a87544371ae7
>2012-12-17 10:03:21,668 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5b398282] spmStart polling ended: taskId = 7926acf1-8885-40a0-b9df-a87544371ae7 task status = finished
>2012-12-17 10:03:21,686 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5b398282] spmStart polling ended, spm status: SPM
>2012-12-17 10:03:21,687 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [5b398282] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7926acf1-8885-40a0-b9df-a87544371ae7), log id: 9b58fe1
>2012-12-17 10:03:21,705 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [5b398282] FINISH, HSMClearTaskVDSCommand, log id: 9b58fe1
>2012-12-17 10:03:21,706 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5b398282] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6ef901c1, log id: 65ffd80d
>2012-12-17 10:03:21,807 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5b398282] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:03:21,990 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-73) [786860d5] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:03:21,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 16a169a3
>2012-12-17 10:03:22,165 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5b398282] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:03:22,625 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [62ee0772] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:03:22,781 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 65ca6b14
>2012-12-17 10:03:22,783 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2e146a85
>2012-12-17 10:03:22,803 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:03:23,338 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] FINISH, SpmStopVDSCommand, log id: 2e146a85
>2012-12-17 10:03:23,366 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] FINISH, ResetIrsVDSCommand, log id: 65ca6b14
>2012-12-17 10:03:23,377 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 31d70570
>2012-12-17 10:03:23,377 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [62ee0772] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:03:23,377 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 31d70570
>2012-12-17 10:03:23,515 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [786860d5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:03:23,547 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [786860d5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5037
>2012-12-17 10:03:23,549 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5037, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 69fc1c91
>2012-12-17 10:03:23,571 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [786860d5] spmStart polling started: taskId = aad38a30-e888-4ed0-945c-33c4bb844f83
>2012-12-17 10:03:24,592 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [786860d5] spmStart polling ended: taskId = aad38a30-e888-4ed0-945c-33c4bb844f83 task status = finished
>2012-12-17 10:03:24,610 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [786860d5] spmStart polling ended, spm status: SPM
>2012-12-17 10:03:24,612 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=aad38a30-e888-4ed0-945c-33c4bb844f83), log id: 6c0e61e1
>2012-12-17 10:03:24,630 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, HSMClearTaskVDSCommand, log id: 6c0e61e1
>2012-12-17 10:03:24,630 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@243c3233, log id: 69fc1c91
>2012-12-17 10:03:24,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [786860d5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:03:24,689 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:03:24,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 746983f0
>2012-12-17 10:03:24,712 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 746983f0
>2012-12-17 10:03:24,712 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 16a169a3
>2012-12-17 10:03:24,713 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-73) [786860d5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:03:24,713 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3b553d45
>2012-12-17 10:03:24,713 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:03:24,714 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 796072b7
>2012-12-17 10:03:24,732 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 796072b7
>2012-12-17 10:03:24,732 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3b553d45
>2012-12-17 10:03:24,732 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-73) [786860d5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:03:32,268 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1ecb2997] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:03:33,795 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [193f8ef] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:03:34,187 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [53480d94] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:03:34,487 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: eadacb1
>2012-12-17 10:03:34,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 481a6b49
>2012-12-17 10:03:34,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [53480d94] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:03:34,672 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, SpmStopVDSCommand, log id: 481a6b49
>2012-12-17 10:03:34,748 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, ResetIrsVDSCommand, log id: eadacb1
>2012-12-17 10:03:34,759 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5812c372
>2012-12-17 10:03:34,759 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [53480d94] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:03:34,759 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5812c372
>2012-12-17 10:03:42,583 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:03:44,961 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:03:45,033 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5038
>2012-12-17 10:03:45,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5038, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7cc0282d
>2012-12-17 10:03:45,062 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] spmStart polling started: taskId = 72ce5fb8-b929-4316-8407-bd9b0b4bec63
>2012-12-17 10:03:46,086 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] spmStart polling ended: taskId = 72ce5fb8-b929-4316-8407-bd9b0b4bec63 task status = finished
>2012-12-17 10:03:46,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] spmStart polling ended, spm status: SPM
>2012-12-17 10:03:46,105 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=72ce5fb8-b929-4316-8407-bd9b0b4bec63), log id: 30fa6f76
>2012-12-17 10:03:46,124 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, HSMClearTaskVDSCommand, log id: 30fa6f76
>2012-12-17 10:03:46,124 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@9abc519, log id: 7cc0282d
>2012-12-17 10:03:46,945 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:03:47,277 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5fb78147] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:03:47,277 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1a5a2ded
>2012-12-17 10:03:47,304 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:03:47,809 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [2f07235e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:03:47,920 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [2f07235e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1a8fc6ba
>2012-12-17 10:03:47,921 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [2f07235e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 35ba84ec
>2012-12-17 10:03:47,941 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [2f07235e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:03:48,063 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [2f07235e] FINISH, SpmStopVDSCommand, log id: 35ba84ec
>2012-12-17 10:03:48,107 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [2f07235e] FINISH, ResetIrsVDSCommand, log id: 1a8fc6ba
>2012-12-17 10:03:48,118 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [2f07235e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6596308f
>2012-12-17 10:03:48,118 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [2f07235e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:03:48,118 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [2f07235e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6596308f
>2012-12-17 10:03:48,262 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:03:48,276 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5039
>2012-12-17 10:03:48,277 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5039, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5f44f41e
>2012-12-17 10:03:48,302 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling started: taskId = a248f873-dd14-448a-9108-eb4c4598959d
>2012-12-17 10:03:49,322 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling ended: taskId = a248f873-dd14-448a-9108-eb4c4598959d task status = finished
>2012-12-17 10:03:49,341 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling ended, spm status: SPM
>2012-12-17 10:03:49,342 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a248f873-dd14-448a-9108-eb4c4598959d), log id: 6c3b8a1f
>2012-12-17 10:03:49,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, HSMClearTaskVDSCommand, log id: 6c3b8a1f
>2012-12-17 10:03:49,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@38e930bc, log id: 5f44f41e
>2012-12-17 10:03:49,450 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:03:49,522 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:03:49,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 222b68a1
>2012-12-17 10:03:49,542 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 222b68a1
>2012-12-17 10:03:49,542 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1a5a2ded
>2012-12-17 10:03:49,542 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5fb78147] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:03:49,543 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4fe2338d
>2012-12-17 10:03:49,543 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:03:49,544 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3b6a1a93
>2012-12-17 10:03:49,563 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3b6a1a93
>2012-12-17 10:03:49,563 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4fe2338d
>2012-12-17 10:03:49,563 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5fb78147] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:03:53,010 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [38047437] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:03:58,328 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1dafbc5c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:03:58,683 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [4b390d42] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:03:58,818 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [4b390d42] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 420c58ce
>2012-12-17 10:03:58,819 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [4b390d42] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 488f09d5
>2012-12-17 10:03:58,839 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [4b390d42] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:03:58,986 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [4b390d42] FINISH, SpmStopVDSCommand, log id: 488f09d5
>2012-12-17 10:03:59,024 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [4b390d42] FINISH, ResetIrsVDSCommand, log id: 420c58ce
>2012-12-17 10:03:59,035 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [4b390d42] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7efb6aee
>2012-12-17 10:03:59,035 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [4b390d42] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:03:59,035 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [4b390d42] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7efb6aee
>2012-12-17 10:04:03,308 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:04:09,110 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [75114a28] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:04:09,118 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [75114a28] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5040
>2012-12-17 10:04:09,128 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [75114a28] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5040, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 21b95bce
>2012-12-17 10:04:09,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [75114a28] spmStart polling started: taskId = aef60ccc-5f7d-4839-85fd-c3d82b093aae
>2012-12-17 10:04:10,170 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [75114a28] spmStart polling ended: taskId = aef60ccc-5f7d-4839-85fd-c3d82b093aae task status = finished
>2012-12-17 10:04:10,188 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [75114a28] spmStart polling ended, spm status: SPM
>2012-12-17 10:04:10,190 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [75114a28] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=aef60ccc-5f7d-4839-85fd-c3d82b093aae), log id: 59e6eed8
>2012-12-17 10:04:10,208 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [75114a28] FINISH, HSMClearTaskVDSCommand, log id: 59e6eed8
>2012-12-17 10:04:10,208 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [75114a28] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@484ad504, log id: 21b95bce
>2012-12-17 10:04:10,220 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [75114a28] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:04:10,253 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [5920de03] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:04:10,253 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 40437d43
>2012-12-17 10:04:10,286 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [75114a28] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:04:11,838 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [10600189] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:04:12,633 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 25b58411
>2012-12-17 10:04:12,635 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4ed6122
>2012-12-17 10:04:12,655 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [10600189] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:04:12,806 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, SpmStopVDSCommand, log id: 4ed6122
>2012-12-17 10:04:12,832 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, ResetIrsVDSCommand, log id: 25b58411
>2012-12-17 10:04:12,844 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 385241f4
>2012-12-17 10:04:12,844 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:04:12,844 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 385241f4
>2012-12-17 10:04:12,895 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [5920de03] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:04:12,907 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [5920de03] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5041
>2012-12-17 10:04:12,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5041, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 71628853
>2012-12-17 10:04:12,929 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [5920de03] spmStart polling started: taskId = c6dae77d-e1a6-4ffa-bc3c-d49a2a8c59c4
>2012-12-17 10:04:13,963 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [5920de03] spmStart polling ended: taskId = c6dae77d-e1a6-4ffa-bc3c-d49a2a8c59c4 task status = finished
>2012-12-17 10:04:13,980 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [5920de03] spmStart polling ended, spm status: SPM
>2012-12-17 10:04:13,981 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c6dae77d-e1a6-4ffa-bc3c-d49a2a8c59c4), log id: 5ea03201
>2012-12-17 10:04:13,999 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, HSMClearTaskVDSCommand, log id: 5ea03201
>2012-12-17 10:04:13,999 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6733102f, log id: 71628853
>2012-12-17 10:04:14,012 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [5920de03] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:04:14,031 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:04:14,033 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 65ef10bc
>2012-12-17 10:04:14,052 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 65ef10bc
>2012-12-17 10:04:14,052 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 40437d43
>2012-12-17 10:04:14,052 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [5920de03] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:04:14,052 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3233daec
>2012-12-17 10:04:14,052 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:04:14,053 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d760880
>2012-12-17 10:04:14,071 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d760880
>2012-12-17 10:04:14,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3233daec
>2012-12-17 10:04:14,071 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [5920de03] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:04:22,970 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:04:23,110 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [3c3f7918] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:04:23,178 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [3c3f7918] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 649efde7
>2012-12-17 10:04:23,179 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [3c3f7918] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4ddf3d72
>2012-12-17 10:04:23,200 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [3c3f7918] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:04:23,371 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [3c3f7918] FINISH, SpmStopVDSCommand, log id: 4ddf3d72
>2012-12-17 10:04:23,389 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [3c3f7918] FINISH, ResetIrsVDSCommand, log id: 649efde7
>2012-12-17 10:04:23,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [3c3f7918] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 20ee4f5a
>2012-12-17 10:04:23,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [3c3f7918] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:04:23,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [3c3f7918] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 20ee4f5a
>2012-12-17 10:04:23,799 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [4a8b829d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:04:33,505 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6edf3aeb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:04:33,565 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6edf3aeb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5042
>2012-12-17 10:04:33,566 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5042, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 584f8a32
>2012-12-17 10:04:33,590 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] spmStart polling started: taskId = 9728bcdc-475d-4a21-9b07-3d9b701b8d90
>2012-12-17 10:04:34,611 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] spmStart polling ended: taskId = 9728bcdc-475d-4a21-9b07-3d9b701b8d90 task status = finished
>2012-12-17 10:04:34,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] spmStart polling ended, spm status: SPM
>2012-12-17 10:04:34,630 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9728bcdc-475d-4a21-9b07-3d9b701b8d90), log id: 503d1f56
>2012-12-17 10:04:34,648 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] FINISH, HSMClearTaskVDSCommand, log id: 503d1f56
>2012-12-17 10:04:34,648 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5cd47041, log id: 584f8a32
>2012-12-17 10:04:34,708 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6edf3aeb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:04:34,804 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [26a9278b] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:04:34,804 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2e486d93
>2012-12-17 10:04:34,844 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6edf3aeb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:04:35,139 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-22) [42fa3e4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:04:35,323 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 13030497
>2012-12-17 10:04:35,325 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: ff7d99c
>2012-12-17 10:04:35,344 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:04:35,927 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, SpmStopVDSCommand, log id: ff7d99c
>2012-12-17 10:04:37,004 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, ResetIrsVDSCommand, log id: 13030497
>2012-12-17 10:04:37,015 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6b95e719
>2012-12-17 10:04:37,015 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [42fa3e4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:04:37,015 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6b95e719
>2012-12-17 10:04:37,351 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [26a9278b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:04:37,430 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [26a9278b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5043
>2012-12-17 10:04:37,431 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5043, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2ae1a24b
>2012-12-17 10:04:37,454 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] spmStart polling started: taskId = f7d5661c-331a-4171-8d37-879254475f3b
>2012-12-17 10:04:38,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] spmStart polling ended: taskId = f7d5661c-331a-4171-8d37-879254475f3b task status = finished
>2012-12-17 10:04:38,493 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] spmStart polling ended, spm status: SPM
>2012-12-17 10:04:38,495 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f7d5661c-331a-4171-8d37-879254475f3b), log id: 3e3b8b5c
>2012-12-17 10:04:38,514 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, HSMClearTaskVDSCommand, log id: 3e3b8b5c
>2012-12-17 10:04:38,514 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@373a64ef, log id: 2ae1a24b
>2012-12-17 10:04:38,557 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [26a9278b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:04:38,648 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:04:38,649 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 43790b0f
>2012-12-17 10:04:38,670 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 43790b0f
>2012-12-17 10:04:38,670 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2e486d93
>2012-12-17 10:04:38,670 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [26a9278b] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:04:38,670 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3fc52864
>2012-12-17 10:04:38,670 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:04:38,671 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 26bb4133
>2012-12-17 10:04:38,690 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 26bb4133
>2012-12-17 10:04:38,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3fc52864
>2012-12-17 10:04:38,690 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [26a9278b] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:04:44,381 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [66369950] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:04:47,450 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [4b390d42] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:04:48,053 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [71a21411] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:04:48,218 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [71a21411] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6f1669bf
>2012-12-17 10:04:48,221 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [71a21411] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 565c1d46
>2012-12-17 10:04:48,240 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [71a21411] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:04:48,452 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [71a21411] FINISH, SpmStopVDSCommand, log id: 565c1d46
>2012-12-17 10:04:48,513 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [71a21411] FINISH, ResetIrsVDSCommand, log id: 6f1669bf
>2012-12-17 10:04:48,524 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [71a21411] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6de14679
>2012-12-17 10:04:48,524 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [71a21411] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:04:48,524 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [71a21411] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6de14679
>2012-12-17 10:04:54,686 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:04:58,682 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [62ee0772] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:04:58,724 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [62ee0772] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5044
>2012-12-17 10:04:58,726 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5044, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b1605a6
>2012-12-17 10:04:58,758 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] spmStart polling started: taskId = cf1054cc-d18c-4791-9f3e-12b621f064ed
>2012-12-17 10:04:59,780 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] spmStart polling ended: taskId = cf1054cc-d18c-4791-9f3e-12b621f064ed task status = finished
>2012-12-17 10:04:59,798 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] spmStart polling ended, spm status: SPM
>2012-12-17 10:04:59,799 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cf1054cc-d18c-4791-9f3e-12b621f064ed), log id: 208e1d14
>2012-12-17 10:04:59,818 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] FINISH, HSMClearTaskVDSCommand, log id: 208e1d14
>2012-12-17 10:04:59,818 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@52171cbe, log id: 6b1605a6
>2012-12-17 10:04:59,841 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [62ee0772] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:04:59,874 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [10600189] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:04:59,874 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 347abb03
>2012-12-17 10:04:59,898 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [62ee0772] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:05:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Checking autorecoverable hosts
>2012-12-17 10:05:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Autorecovering 0 hosts
>2012-12-17 10:05:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Checking autorecoverable hosts done
>2012-12-17 10:05:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Autorecovering storage domains is disabled, skipping
>2012-12-17 10:05:00,021 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [3d52876a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:05:00,072 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [3d52876a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2232d05a
>2012-12-17 10:05:00,073 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [3d52876a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3eb85cb9
>2012-12-17 10:05:00,093 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [3d52876a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:00,242 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [3d52876a] FINISH, SpmStopVDSCommand, log id: 3eb85cb9
>2012-12-17 10:05:00,260 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [3d52876a] FINISH, ResetIrsVDSCommand, log id: 2232d05a
>2012-12-17 10:05:00,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [3d52876a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 785986ee
>2012-12-17 10:05:00,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3d52876a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:00,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [3d52876a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 785986ee
>2012-12-17 10:05:00,329 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:05:00,342 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5045
>2012-12-17 10:05:00,344 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5045, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 701101df
>2012-12-17 10:05:00,367 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] spmStart polling started: taskId = c760556c-239f-4fb5-9969-3c164b95adb4
>2012-12-17 10:05:01,389 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] spmStart polling ended: taskId = c760556c-239f-4fb5-9969-3c164b95adb4 task status = finished
>2012-12-17 10:05:01,407 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] spmStart polling ended, spm status: SPM
>2012-12-17 10:05:01,408 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c760556c-239f-4fb5-9969-3c164b95adb4), log id: 5e42bbf9
>2012-12-17 10:05:01,426 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, HSMClearTaskVDSCommand, log id: 5e42bbf9
>2012-12-17 10:05:01,426 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6f4308c7, log id: 701101df
>2012-12-17 10:05:01,488 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:05:01,683 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:01,684 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 397b5197
>2012-12-17 10:05:01,705 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 397b5197
>2012-12-17 10:05:01,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 347abb03
>2012-12-17 10:05:01,706 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [10600189] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:05:01,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4f29adfc
>2012-12-17 10:05:01,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:01,707 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: bf5e032
>2012-12-17 10:05:01,726 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: bf5e032
>2012-12-17 10:05:01,726 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4f29adfc
>2012-12-17 10:05:01,726 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [10600189] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:05:04,908 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [69818e3e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:05:10,390 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [3c3f7918] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:05:10,543 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [8b964cc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:05:10,619 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4f974438
>2012-12-17 10:05:10,621 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 490255cd
>2012-12-17 10:05:10,641 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:10,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] FINISH, SpmStopVDSCommand, log id: 490255cd
>2012-12-17 10:05:10,878 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] FINISH, ResetIrsVDSCommand, log id: 4f974438
>2012-12-17 10:05:10,889 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 678f7f41
>2012-12-17 10:05:10,889 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [8b964cc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:10,889 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 678f7f41
>2012-12-17 10:05:15,126 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:05:21,067 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:05:21,379 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5046
>2012-12-17 10:05:21,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5046, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3f04bdd1
>2012-12-17 10:05:21,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] spmStart polling started: taskId = 4fd3b1cf-950e-49a9-ba67-c67f136b833b
>2012-12-17 10:05:22,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] spmStart polling ended: taskId = 4fd3b1cf-950e-49a9-ba67-c67f136b833b task status = finished
>2012-12-17 10:05:22,446 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] spmStart polling ended, spm status: SPM
>2012-12-17 10:05:22,447 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4fd3b1cf-950e-49a9-ba67-c67f136b833b), log id: 2bcf0e67
>2012-12-17 10:05:22,465 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] FINISH, HSMClearTaskVDSCommand, log id: 2bcf0e67
>2012-12-17 10:05:22,465 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@26baa2a9, log id: 3f04bdd1
>2012-12-17 10:05:22,534 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:05:22,647 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [20d09da7] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:05:22,647 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 69767428
>2012-12-17 10:05:22,736 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:05:22,876 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-43) [b0e3bca] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:05:22,927 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3d4ed90a
>2012-12-17 10:05:22,929 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 46149bc6
>2012-12-17 10:05:22,948 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:23,367 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, SpmStopVDSCommand, log id: 46149bc6
>2012-12-17 10:05:23,382 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, ResetIrsVDSCommand, log id: 3d4ed90a
>2012-12-17 10:05:23,393 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 319a9f16
>2012-12-17 10:05:23,393 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [b0e3bca] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:23,393 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 319a9f16
>2012-12-17 10:05:23,456 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [20d09da7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:05:23,468 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [20d09da7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5047
>2012-12-17 10:05:23,470 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5047, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 74106d05
>2012-12-17 10:05:23,492 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] spmStart polling started: taskId = 1828dcec-a632-4bce-95b4-2c0ef99be33d
>2012-12-17 10:05:24,514 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] spmStart polling ended: taskId = 1828dcec-a632-4bce-95b4-2c0ef99be33d task status = finished
>2012-12-17 10:05:24,533 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] spmStart polling ended, spm status: SPM
>2012-12-17 10:05:24,534 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1828dcec-a632-4bce-95b4-2c0ef99be33d), log id: 32963dab
>2012-12-17 10:05:24,553 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, HSMClearTaskVDSCommand, log id: 32963dab
>2012-12-17 10:05:24,553 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@25aabf7f, log id: 74106d05
>2012-12-17 10:05:24,583 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [20d09da7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:05:24,600 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:24,603 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1fc586f4
>2012-12-17 10:05:24,624 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1fc586f4
>2012-12-17 10:05:24,624 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 69767428
>2012-12-17 10:05:24,624 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [20d09da7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:05:24,624 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3dad3171
>2012-12-17 10:05:24,624 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:24,625 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3d900aa2
>2012-12-17 10:05:24,644 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3d900aa2
>2012-12-17 10:05:24,644 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3dad3171
>2012-12-17 10:05:24,644 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [20d09da7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:05:25,320 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:05:33,501 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [415bb057] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:05:33,613 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-35) [464dfbdd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:05:33,689 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 53f45987
>2012-12-17 10:05:33,690 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1aaeda01
>2012-12-17 10:05:33,710 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:34,272 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] FINISH, SpmStopVDSCommand, log id: 1aaeda01
>2012-12-17 10:05:34,286 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] FINISH, ResetIrsVDSCommand, log id: 53f45987
>2012-12-17 10:05:34,297 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5d02ac9e
>2012-12-17 10:05:34,297 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [464dfbdd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:34,297 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5d02ac9e
>2012-12-17 10:05:35,501 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:05:44,344 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:05:44,375 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5048
>2012-12-17 10:05:44,376 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5048, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 284c17aa
>2012-12-17 10:05:44,399 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] spmStart polling started: taskId = 72c63a02-5d82-4fd1-8067-bc5dae55bc8b
>2012-12-17 10:05:45,421 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] spmStart polling ended: taskId = 72c63a02-5d82-4fd1-8067-bc5dae55bc8b task status = finished
>2012-12-17 10:05:45,439 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] spmStart polling ended, spm status: SPM
>2012-12-17 10:05:45,441 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=72c63a02-5d82-4fd1-8067-bc5dae55bc8b), log id: 340fb222
>2012-12-17 10:05:45,459 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] FINISH, HSMClearTaskVDSCommand, log id: 340fb222
>2012-12-17 10:05:45,459 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7277f0e1, log id: 284c17aa
>2012-12-17 10:05:45,505 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:05:45,548 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:05:45,548 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6a2ac029
>2012-12-17 10:05:45,567 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:05:45,789 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [375e73d3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:05:45,856 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [375e73d3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 661ac216
>2012-12-17 10:05:45,857 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [375e73d3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 190daf6b
>2012-12-17 10:05:45,877 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [375e73d3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:46,029 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [375e73d3] FINISH, SpmStopVDSCommand, log id: 190daf6b
>2012-12-17 10:05:46,555 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [375e73d3] FINISH, ResetIrsVDSCommand, log id: 661ac216
>2012-12-17 10:05:46,566 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [375e73d3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3f384a39
>2012-12-17 10:05:46,566 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [375e73d3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:46,566 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [375e73d3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3f384a39
>2012-12-17 10:05:46,721 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:05:46,766 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5049
>2012-12-17 10:05:46,768 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5049, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 756e85f5
>2012-12-17 10:05:46,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling started: taskId = 8cf4c681-55f2-4315-b0b0-24f756854e0b
>2012-12-17 10:05:47,811 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended: taskId = 8cf4c681-55f2-4315-b0b0-24f756854e0b task status = finished
>2012-12-17 10:05:47,829 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended, spm status: SPM
>2012-12-17 10:05:47,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8cf4c681-55f2-4315-b0b0-24f756854e0b), log id: 7916c3d5
>2012-12-17 10:05:47,848 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMClearTaskVDSCommand, log id: 7916c3d5
>2012-12-17 10:05:47,848 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4593130e, log id: 756e85f5
>2012-12-17 10:05:47,859 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:05:47,892 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:47,893 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 72fe1098
>2012-12-17 10:05:47,912 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 72fe1098
>2012-12-17 10:05:47,912 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6a2ac029
>2012-12-17 10:05:47,912 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:05:47,912 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3f2046e2
>2012-12-17 10:05:47,912 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:47,913 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3cb2457a
>2012-12-17 10:05:47,932 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3cb2457a
>2012-12-17 10:05:47,932 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f2046e2
>2012-12-17 10:05:47,932 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:05:56,908 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [4549d797] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:05:56,909 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [39a6c2d3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:05:57,074 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:05:57,145 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: a3f922c
>2012-12-17 10:05:57,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 636a1ed8
>2012-12-17 10:05:57,170 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:57,971 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] FINISH, SpmStopVDSCommand, log id: 636a1ed8
>2012-12-17 10:05:57,987 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] FINISH, ResetIrsVDSCommand, log id: a3f922c
>2012-12-17 10:05:57,998 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1168a7ec
>2012-12-17 10:05:57,998 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:05:57,998 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1168a7ec
>2012-12-17 10:06:07,477 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:06:08,061 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [21d8114] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:06:08,072 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [21d8114] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5050
>2012-12-17 10:06:08,073 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [21d8114] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5050, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6c6eb586
>2012-12-17 10:06:08,095 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [21d8114] spmStart polling started: taskId = bfbcf4d7-c4a1-4ddf-acab-7882a230856d
>2012-12-17 10:06:09,117 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [21d8114] spmStart polling ended: taskId = bfbcf4d7-c4a1-4ddf-acab-7882a230856d task status = finished
>2012-12-17 10:06:09,135 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [21d8114] spmStart polling ended, spm status: SPM
>2012-12-17 10:06:09,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [21d8114] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bfbcf4d7-c4a1-4ddf-acab-7882a230856d), log id: 5d1737c1
>2012-12-17 10:06:09,154 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [21d8114] FINISH, HSMClearTaskVDSCommand, log id: 5d1737c1
>2012-12-17 10:06:09,154 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [21d8114] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1f4d8fa6, log id: 6c6eb586
>2012-12-17 10:06:09,164 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [21d8114] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:06:09,209 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-37) [24cb4742] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:06:09,209 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 13791afa
>2012-12-17 10:06:09,228 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [21d8114] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:06:09,433 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [18826afe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:06:09,535 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [18826afe] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 48f659b5
>2012-12-17 10:06:09,540 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [18826afe] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3a91240d
>2012-12-17 10:06:09,570 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [18826afe] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:09,761 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [18826afe] FINISH, SpmStopVDSCommand, log id: 3a91240d
>2012-12-17 10:06:09,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [18826afe] FINISH, ResetIrsVDSCommand, log id: 48f659b5
>2012-12-17 10:06:09,841 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [18826afe] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: ad3686f
>2012-12-17 10:06:09,841 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [18826afe] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:09,841 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [18826afe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: ad3686f
>2012-12-17 10:06:09,986 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [24cb4742] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:06:10,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [24cb4742] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5051
>2012-12-17 10:06:10,038 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5051, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2c401734
>2012-12-17 10:06:10,064 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] spmStart polling started: taskId = bcf00db9-5ff6-4438-8c6b-d90a767c61e3
>2012-12-17 10:06:11,596 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] spmStart polling ended: taskId = bcf00db9-5ff6-4438-8c6b-d90a767c61e3 task status = finished
>2012-12-17 10:06:11,614 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] spmStart polling ended, spm status: SPM
>2012-12-17 10:06:11,615 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bcf00db9-5ff6-4438-8c6b-d90a767c61e3), log id: 4d6e6ac1
>2012-12-17 10:06:11,633 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, HSMClearTaskVDSCommand, log id: 4d6e6ac1
>2012-12-17 10:06:11,633 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@c22cc9e, log id: 2c401734
>2012-12-17 10:06:11,661 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [24cb4742] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:06:11,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:11,767 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1972f3ae
>2012-12-17 10:06:11,789 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1972f3ae
>2012-12-17 10:06:11,789 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 13791afa
>2012-12-17 10:06:11,789 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-37) [24cb4742] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:06:11,790 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 49dc318f
>2012-12-17 10:06:11,790 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:11,791 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 13c09d20
>2012-12-17 10:06:11,810 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 13c09d20
>2012-12-17 10:06:11,810 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 49dc318f
>2012-12-17 10:06:11,810 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-37) [24cb4742] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:06:17,715 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:06:20,037 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [6a7edb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:06:20,174 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [729d4e04] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:06:20,217 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [729d4e04] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6ee8dc7
>2012-12-17 10:06:20,218 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [729d4e04] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2748c783
>2012-12-17 10:06:20,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [729d4e04] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:20,383 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [729d4e04] FINISH, SpmStopVDSCommand, log id: 2748c783
>2012-12-17 10:06:20,413 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [729d4e04] FINISH, ResetIrsVDSCommand, log id: 6ee8dc7
>2012-12-17 10:06:20,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [729d4e04] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: e55f833
>2012-12-17 10:06:20,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [729d4e04] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:20,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [729d4e04] FINISH, MarkPoolInReconstructModeVDSCommand, log id: e55f833
>2012-12-17 10:06:27,887 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:06:30,575 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [3c62925a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:06:30,687 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [3c62925a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5052
>2012-12-17 10:06:30,689 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5052, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 226188d7
>2012-12-17 10:06:30,711 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] spmStart polling started: taskId = 81c2f99a-953e-4a75-8d24-be2b4e87cf78
>2012-12-17 10:06:31,733 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] spmStart polling ended: taskId = 81c2f99a-953e-4a75-8d24-be2b4e87cf78 task status = finished
>2012-12-17 10:06:31,752 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] spmStart polling ended, spm status: SPM
>2012-12-17 10:06:31,753 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=81c2f99a-953e-4a75-8d24-be2b4e87cf78), log id: 57774480
>2012-12-17 10:06:31,772 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] FINISH, HSMClearTaskVDSCommand, log id: 57774480
>2012-12-17 10:06:31,772 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@57325c5c, log id: 226188d7
>2012-12-17 10:06:31,793 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [3c62925a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:06:31,830 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [384b399c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:06:31,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6816a4be
>2012-12-17 10:06:31,850 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [3c62925a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:06:32,089 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [5cd90aa7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:06:32,161 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [5cd90aa7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 26c5d740
>2012-12-17 10:06:32,162 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [5cd90aa7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 57da04a1
>2012-12-17 10:06:32,182 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [5cd90aa7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:32,325 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [5cd90aa7] FINISH, SpmStopVDSCommand, log id: 57da04a1
>2012-12-17 10:06:32,385 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [5cd90aa7] FINISH, ResetIrsVDSCommand, log id: 26c5d740
>2012-12-17 10:06:32,396 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [5cd90aa7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e1c41cd
>2012-12-17 10:06:32,396 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [5cd90aa7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:32,396 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [5cd90aa7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e1c41cd
>2012-12-17 10:06:32,443 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:06:32,454 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5053
>2012-12-17 10:06:32,456 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5053, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 46276be
>2012-12-17 10:06:32,478 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling started: taskId = b835766e-e65d-45eb-89bc-b42f36159d0b
>2012-12-17 10:06:33,499 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling ended: taskId = b835766e-e65d-45eb-89bc-b42f36159d0b task status = finished
>2012-12-17 10:06:33,517 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling ended, spm status: SPM
>2012-12-17 10:06:33,518 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b835766e-e65d-45eb-89bc-b42f36159d0b), log id: 2a1658a9
>2012-12-17 10:06:33,537 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, HSMClearTaskVDSCommand, log id: 2a1658a9
>2012-12-17 10:06:33,537 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2116d4d5, log id: 46276be
>2012-12-17 10:06:33,554 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:06:33,568 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:33,570 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2c1a1da6
>2012-12-17 10:06:33,589 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2c1a1da6
>2012-12-17 10:06:33,589 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6816a4be
>2012-12-17 10:06:33,589 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [384b399c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:06:33,589 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1dde98f3
>2012-12-17 10:06:33,589 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:33,591 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1462b6ca
>2012-12-17 10:06:33,609 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1462b6ca
>2012-12-17 10:06:33,609 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1dde98f3
>2012-12-17 10:06:33,609 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [384b399c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:06:38,075 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [78864454] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:06:42,488 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [3e0f1a8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:06:42,589 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-54) [75689c2c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:06:42,644 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: c56637f
>2012-12-17 10:06:42,645 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6fe5701e
>2012-12-17 10:06:42,665 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:42,966 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, SpmStopVDSCommand, log id: 6fe5701e
>2012-12-17 10:06:42,979 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, ResetIrsVDSCommand, log id: c56637f
>2012-12-17 10:06:42,989 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 27b3ea9d
>2012-12-17 10:06:42,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:42,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 27b3ea9d
>2012-12-17 10:06:48,248 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:06:53,036 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:06:53,052 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5054
>2012-12-17 10:06:53,053 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5054, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1d07078d
>2012-12-17 10:06:53,076 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] spmStart polling started: taskId = 6682a771-d5b1-4cf6-9c07-7e6524d98ad5
>2012-12-17 10:06:54,098 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] spmStart polling ended: taskId = 6682a771-d5b1-4cf6-9c07-7e6524d98ad5 task status = finished
>2012-12-17 10:06:54,116 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] spmStart polling ended, spm status: SPM
>2012-12-17 10:06:54,118 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6682a771-d5b1-4cf6-9c07-7e6524d98ad5), log id: 56b962b1
>2012-12-17 10:06:54,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, HSMClearTaskVDSCommand, log id: 56b962b1
>2012-12-17 10:06:54,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@24fe7ef2, log id: 1d07078d
>2012-12-17 10:06:54,150 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:06:54,186 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:06:54,186 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 51ff7820
>2012-12-17 10:06:54,208 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:06:54,335 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-60) [5074b866] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:06:54,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [5074b866] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5ad2bb40
>2012-12-17 10:06:54,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [5074b866] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5f79c013
>2012-12-17 10:06:54,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [5074b866] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:54,533 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [5074b866] FINISH, SpmStopVDSCommand, log id: 5f79c013
>2012-12-17 10:06:54,543 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [5074b866] FINISH, ResetIrsVDSCommand, log id: 5ad2bb40
>2012-12-17 10:06:54,554 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [5074b866] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 13d5f1d7
>2012-12-17 10:06:54,554 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [5074b866] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:54,554 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [5074b866] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 13d5f1d7
>2012-12-17 10:06:54,603 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:06:54,613 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5055
>2012-12-17 10:06:54,614 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5055, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 71fdb627
>2012-12-17 10:06:54,636 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling started: taskId = 589ac769-24ec-49fc-b320-9532618da2fa
>2012-12-17 10:06:55,658 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling ended: taskId = 589ac769-24ec-49fc-b320-9532618da2fa task status = finished
>2012-12-17 10:06:55,677 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling ended, spm status: SPM
>2012-12-17 10:06:55,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=589ac769-24ec-49fc-b320-9532618da2fa), log id: 1eca421c
>2012-12-17 10:06:55,696 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMClearTaskVDSCommand, log id: 1eca421c
>2012-12-17 10:06:55,696 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@70ba4c0e, log id: 71fdb627
>2012-12-17 10:06:55,722 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:06:55,732 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:55,735 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: d47d501
>2012-12-17 10:06:55,755 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: d47d501
>2012-12-17 10:06:55,755 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 51ff7820
>2012-12-17 10:06:55,755 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:06:55,756 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6776f4c6
>2012-12-17 10:06:55,756 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:06:55,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1cab66a7
>2012-12-17 10:06:55,775 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1cab66a7
>2012-12-17 10:06:55,775 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6776f4c6
>2012-12-17 10:06:55,775 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:06:58,463 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:07:04,643 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:07:04,778 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-100) [6001c862] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:07:04,826 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [6001c862] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2e2cc9a4
>2012-12-17 10:07:04,827 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [6001c862] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6aa77541
>2012-12-17 10:07:04,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [6001c862] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:07:05,013 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [6001c862] FINISH, SpmStopVDSCommand, log id: 6aa77541
>2012-12-17 10:07:05,032 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [6001c862] FINISH, ResetIrsVDSCommand, log id: 2e2cc9a4
>2012-12-17 10:07:05,044 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [6001c862] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 30162110
>2012-12-17 10:07:05,044 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6001c862] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:07:05,044 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [6001c862] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 30162110
>2012-12-17 10:07:08,694 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4d407d58] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:07:15,090 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [511011e7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:07:15,109 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [511011e7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5056
>2012-12-17 10:07:15,111 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [511011e7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5056, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3b42a2b
>2012-12-17 10:07:15,135 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [511011e7] spmStart polling started: taskId = 9b32d14a-2a4a-4e02-a8a1-2a4433e2e543
>2012-12-17 10:07:16,158 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [511011e7] spmStart polling ended: taskId = 9b32d14a-2a4a-4e02-a8a1-2a4433e2e543 task status = finished
>2012-12-17 10:07:16,177 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [511011e7] spmStart polling ended, spm status: SPM
>2012-12-17 10:07:16,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [511011e7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9b32d14a-2a4a-4e02-a8a1-2a4433e2e543), log id: 715ee6ce
>2012-12-17 10:07:16,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [511011e7] FINISH, HSMClearTaskVDSCommand, log id: 715ee6ce
>2012-12-17 10:07:16,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [511011e7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@d831888, log id: 3b42a2b
>2012-12-17 10:07:16,479 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [511011e7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:07:16,638 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [56592212] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:07:16,638 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 68e34d84
>2012-12-17 10:07:16,678 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [511011e7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:07:16,896 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [62b3971a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:07:16,942 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [62b3971a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 281576f3
>2012-12-17 10:07:16,944 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [62b3971a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e88b361
>2012-12-17 10:07:16,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [62b3971a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:07:17,135 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [62b3971a] FINISH, SpmStopVDSCommand, log id: 7e88b361
>2012-12-17 10:07:17,174 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [62b3971a] FINISH, ResetIrsVDSCommand, log id: 281576f3
>2012-12-17 10:07:17,189 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [62b3971a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ed9a399
>2012-12-17 10:07:17,189 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [62b3971a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:07:17,189 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [62b3971a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ed9a399
>2012-12-17 10:07:17,266 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [56592212] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:07:17,285 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [56592212] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5057
>2012-12-17 10:07:17,286 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5057, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5869d64a
>2012-12-17 10:07:17,308 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [56592212] spmStart polling started: taskId = 0c714cec-42f4-473a-8860-6dc2a43efae6
>2012-12-17 10:07:18,329 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [56592212] spmStart polling ended: taskId = 0c714cec-42f4-473a-8860-6dc2a43efae6 task status = finished
>2012-12-17 10:07:18,348 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [56592212] spmStart polling ended, spm status: SPM
>2012-12-17 10:07:18,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0c714cec-42f4-473a-8860-6dc2a43efae6), log id: 35f90494
>2012-12-17 10:07:18,367 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, HSMClearTaskVDSCommand, log id: 35f90494
>2012-12-17 10:07:18,367 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6f9bac3a, log id: 5869d64a
>2012-12-17 10:07:18,388 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [56592212] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:07:18,411 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:07:18,413 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 29a377c4
>2012-12-17 10:07:18,431 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 29a377c4
>2012-12-17 10:07:18,432 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 68e34d84
>2012-12-17 10:07:18,432 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [56592212] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:07:18,432 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1f5846f5
>2012-12-17 10:07:18,432 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:07:18,433 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25ff0ea4
>2012-12-17 10:07:18,451 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25ff0ea4
>2012-12-17 10:07:18,451 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1f5846f5
>2012-12-17 10:07:18,451 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [56592212] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:07:18,961 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [5cd90aa7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:07:27,309 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [2c358b5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:07:27,403 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [1db2b76] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:07:27,455 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4958edb6
>2012-12-17 10:07:27,457 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 123a13b9
>2012-12-17 10:07:27,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:07:27,656 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] FINISH, SpmStopVDSCommand, log id: 123a13b9
>2012-12-17 10:07:27,691 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] FINISH, ResetIrsVDSCommand, log id: 4958edb6
>2012-12-17 10:07:27,702 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2beb2e21
>2012-12-17 10:07:27,702 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [1db2b76] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:07:27,702 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2beb2e21
>2012-12-17 10:07:29,183 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [5117cf2d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:07:37,896 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2669bad8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:07:37,907 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2669bad8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5058
>2012-12-17 10:07:37,909 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5058, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 406c7b93
>2012-12-17 10:07:37,931 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] spmStart polling started: taskId = 5789224d-ce39-43a6-bad4-d9ef62a04802
>2012-12-17 10:07:38,953 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] spmStart polling ended: taskId = 5789224d-ce39-43a6-bad4-d9ef62a04802 task status = finished
>2012-12-17 10:07:38,975 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] spmStart polling ended, spm status: SPM
>2012-12-17 10:07:38,977 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5789224d-ce39-43a6-bad4-d9ef62a04802), log id: ea3f1b5
>2012-12-17 10:07:38,995 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] FINISH, HSMClearTaskVDSCommand, log id: ea3f1b5
>2012-12-17 10:07:38,995 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2d51a995, log id: 406c7b93
>2012-12-17 10:07:39,035 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2669bad8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:07:39,118 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [5f725bc5] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:07:39,118 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7715af99
>2012-12-17 10:07:39,135 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2669bad8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:07:39,512 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [6018e3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:07:39,585 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [6018e3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17d73da9
>2012-12-17 10:07:39,586 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [6018e3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5dd8725b
>2012-12-17 10:07:39,606 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [6018e3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:07:39,768 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [6018e3] FINISH, SpmStopVDSCommand, log id: 5dd8725b
>2012-12-17 10:07:39,785 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [6018e3] FINISH, ResetIrsVDSCommand, log id: 17d73da9
>2012-12-17 10:07:39,796 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [6018e3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 651071a1
>2012-12-17 10:07:39,796 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [6018e3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:07:39,796 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [6018e3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 651071a1
>2012-12-17 10:07:39,852 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:07:39,860 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5059
>2012-12-17 10:07:39,861 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5059, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1dd9b5e4
>2012-12-17 10:07:39,882 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] spmStart polling started: taskId = ad6bdd7a-4d68-4dd0-b83c-e7c63ad1ee6c
>2012-12-17 10:07:40,906 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] spmStart polling ended: taskId = ad6bdd7a-4d68-4dd0-b83c-e7c63ad1ee6c task status = finished
>2012-12-17 10:07:40,925 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] spmStart polling ended, spm status: SPM
>2012-12-17 10:07:40,927 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ad6bdd7a-4d68-4dd0-b83c-e7c63ad1ee6c), log id: 12bc8f78
>2012-12-17 10:07:40,945 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, HSMClearTaskVDSCommand, log id: 12bc8f78
>2012-12-17 10:07:40,945 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@196f18e4, log id: 1dd9b5e4
>2012-12-17 10:07:41,527 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:07:41,770 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:07:41,774 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 135cf34c
>2012-12-17 10:07:41,793 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 135cf34c
>2012-12-17 10:07:41,793 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7715af99
>2012-12-17 10:07:41,793 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [5f725bc5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:07:41,794 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3b690c82
>2012-12-17 10:07:41,794 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:07:41,795 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6ec6a2c4
>2012-12-17 10:07:41,813 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6ec6a2c4
>2012-12-17 10:07:41,813 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3b690c82
>2012-12-17 10:07:41,813 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [5f725bc5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:07:49,589 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:07:49,890 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:07:49,999 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-18) [6038c0b1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:07:50,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [6038c0b1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 672fe7e3
>2012-12-17 10:07:50,048 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [6038c0b1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 42ce78af
>2012-12-17 10:07:50,069 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [6038c0b1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:07:50,270 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [6038c0b1] FINISH, SpmStopVDSCommand, log id: 42ce78af
>2012-12-17 10:07:50,284 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [6038c0b1] FINISH, ResetIrsVDSCommand, log id: 672fe7e3
>2012-12-17 10:07:50,295 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [6038c0b1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 43a8d6a9
>2012-12-17 10:07:50,295 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [6038c0b1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:07:50,295 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [6038c0b1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 43a8d6a9
>2012-12-17 10:07:59,774 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:08:00,359 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [71a21411] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:08:00,381 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [71a21411] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5060
>2012-12-17 10:08:00,383 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [71a21411] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5060, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1eeb9318
>2012-12-17 10:08:00,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [71a21411] spmStart polling started: taskId = caf69730-de32-4f15-94f7-346277a49cb5
>2012-12-17 10:08:01,433 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [71a21411] spmStart polling ended: taskId = caf69730-de32-4f15-94f7-346277a49cb5 task status = finished
>2012-12-17 10:08:01,452 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [71a21411] spmStart polling ended, spm status: SPM
>2012-12-17 10:08:01,453 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [71a21411] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=caf69730-de32-4f15-94f7-346277a49cb5), log id: 65624a0c
>2012-12-17 10:08:01,472 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [71a21411] FINISH, HSMClearTaskVDSCommand, log id: 65624a0c
>2012-12-17 10:08:01,472 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [71a21411] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@17eb8dd4, log id: 1eeb9318
>2012-12-17 10:08:01,598 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [71a21411] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:08:01,874 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [21122b75] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:08:01,874 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 203403ef
>2012-12-17 10:08:01,902 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [71a21411] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:08:02,069 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:08:02,124 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 22d47806
>2012-12-17 10:08:02,125 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 23f7386f
>2012-12-17 10:08:02,145 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:02,290 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] FINISH, SpmStopVDSCommand, log id: 23f7386f
>2012-12-17 10:08:02,348 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] FINISH, ResetIrsVDSCommand, log id: 22d47806
>2012-12-17 10:08:02,360 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c5f1e77
>2012-12-17 10:08:02,360 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:02,360 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c5f1e77
>2012-12-17 10:08:02,448 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:08:02,461 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5061
>2012-12-17 10:08:02,463 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5061, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2638ac5e
>2012-12-17 10:08:02,485 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] spmStart polling started: taskId = b2c233c1-bec2-4eca-b43c-3bce38107eb9
>2012-12-17 10:08:03,510 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] spmStart polling ended: taskId = b2c233c1-bec2-4eca-b43c-3bce38107eb9 task status = finished
>2012-12-17 10:08:03,535 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] spmStart polling ended, spm status: SPM
>2012-12-17 10:08:03,537 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b2c233c1-bec2-4eca-b43c-3bce38107eb9), log id: 6b5d619c
>2012-12-17 10:08:03,555 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, HSMClearTaskVDSCommand, log id: 6b5d619c
>2012-12-17 10:08:03,555 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3fb37fd5, log id: 2638ac5e
>2012-12-17 10:08:03,568 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:08:03,580 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:03,585 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6cb8ca1a
>2012-12-17 10:08:03,604 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6cb8ca1a
>2012-12-17 10:08:03,604 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 203403ef
>2012-12-17 10:08:03,604 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [21122b75] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:08:03,605 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 57c936bf
>2012-12-17 10:08:03,605 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:03,606 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2d8422ac
>2012-12-17 10:08:03,624 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2d8422ac
>2012-12-17 10:08:03,624 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 57c936bf
>2012-12-17 10:08:03,624 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [21122b75] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:08:09,987 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [140420a5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:08:12,530 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [5920de03] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:08:12,827 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [40858b1a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:08:12,884 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 77c2d1bb
>2012-12-17 10:08:12,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6f681fcd
>2012-12-17 10:08:12,906 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:13,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] FINISH, SpmStopVDSCommand, log id: 6f681fcd
>2012-12-17 10:08:13,510 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] FINISH, ResetIrsVDSCommand, log id: 77c2d1bb
>2012-12-17 10:08:13,521 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 78bc27f9
>2012-12-17 10:08:13,521 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [40858b1a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:13,521 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 78bc27f9
>2012-12-17 10:08:20,193 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1ecb2997] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:08:23,573 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [8b964cc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:08:23,585 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [8b964cc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5062
>2012-12-17 10:08:23,587 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5062, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 65fa5608
>2012-12-17 10:08:23,608 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] spmStart polling started: taskId = 602db02f-92c1-43a0-982a-774f828f7159
>2012-12-17 10:08:24,629 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] spmStart polling ended: taskId = 602db02f-92c1-43a0-982a-774f828f7159 task status = finished
>2012-12-17 10:08:24,648 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] spmStart polling ended, spm status: SPM
>2012-12-17 10:08:24,649 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=602db02f-92c1-43a0-982a-774f828f7159), log id: 59f7e2fc
>2012-12-17 10:08:24,667 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] FINISH, HSMClearTaskVDSCommand, log id: 59f7e2fc
>2012-12-17 10:08:24,667 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@21573c89, log id: 65fa5608
>2012-12-17 10:08:24,687 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [8b964cc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:08:24,722 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:08:24,722 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 33a4f9e2
>2012-12-17 10:08:24,754 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [8b964cc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:08:24,861 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [a355c3f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:08:24,907 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5032e9d0
>2012-12-17 10:08:24,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f51391c
>2012-12-17 10:08:24,928 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:25,077 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] FINISH, SpmStopVDSCommand, log id: 4f51391c
>2012-12-17 10:08:25,091 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] FINISH, ResetIrsVDSCommand, log id: 5032e9d0
>2012-12-17 10:08:25,102 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 110294cc
>2012-12-17 10:08:25,102 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [a355c3f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:25,102 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 110294cc
>2012-12-17 10:08:25,151 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:08:25,160 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5063
>2012-12-17 10:08:25,162 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5063, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 145e4a8
>2012-12-17 10:08:25,185 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling started: taskId = 06577d4a-8a7b-4684-bba0-8baf007bb20b
>2012-12-17 10:08:26,208 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling ended: taskId = 06577d4a-8a7b-4684-bba0-8baf007bb20b task status = finished
>2012-12-17 10:08:26,227 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling ended, spm status: SPM
>2012-12-17 10:08:26,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=06577d4a-8a7b-4684-bba0-8baf007bb20b), log id: 4a813a1e
>2012-12-17 10:08:26,255 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMClearTaskVDSCommand, log id: 4a813a1e
>2012-12-17 10:08:26,255 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4abde2be, log id: 145e4a8
>2012-12-17 10:08:26,505 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:08:26,815 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:26,817 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5028395f
>2012-12-17 10:08:26,836 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5028395f
>2012-12-17 10:08:26,836 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 33a4f9e2
>2012-12-17 10:08:26,836 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:08:26,836 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: ceb516
>2012-12-17 10:08:26,836 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:26,837 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4e6377de
>2012-12-17 10:08:26,856 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4e6377de
>2012-12-17 10:08:26,856 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: ceb516
>2012-12-17 10:08:26,856 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:08:30,398 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [18826afe] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:08:35,201 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [20d09da7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:08:35,335 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-99) [1290f5b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:08:35,385 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [1290f5b5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: cdcbd9
>2012-12-17 10:08:35,387 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [1290f5b5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 63a11b24
>2012-12-17 10:08:35,408 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [1290f5b5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:36,102 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [1290f5b5] FINISH, SpmStopVDSCommand, log id: 63a11b24
>2012-12-17 10:08:36,625 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [1290f5b5] FINISH, ResetIrsVDSCommand, log id: cdcbd9
>2012-12-17 10:08:36,637 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [1290f5b5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 54552694
>2012-12-17 10:08:36,637 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [1290f5b5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:36,637 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [1290f5b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 54552694
>2012-12-17 10:08:40,587 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:08:46,777 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [470bd088] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:08:46,792 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [470bd088] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5064
>2012-12-17 10:08:46,793 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [470bd088] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5064, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6e5070ea
>2012-12-17 10:08:46,816 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [470bd088] spmStart polling started: taskId = 4b0a00c6-d3a6-47c0-8ba1-df96ccc72c84
>2012-12-17 10:08:47,839 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [470bd088] spmStart polling ended: taskId = 4b0a00c6-d3a6-47c0-8ba1-df96ccc72c84 task status = finished
>2012-12-17 10:08:47,858 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [470bd088] spmStart polling ended, spm status: SPM
>2012-12-17 10:08:47,859 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [470bd088] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4b0a00c6-d3a6-47c0-8ba1-df96ccc72c84), log id: 40b3bf18
>2012-12-17 10:08:47,877 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [470bd088] FINISH, HSMClearTaskVDSCommand, log id: 40b3bf18
>2012-12-17 10:08:47,877 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [470bd088] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@58c6e1d7, log id: 6e5070ea
>2012-12-17 10:08:47,896 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [470bd088] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:08:47,933 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:08:47,934 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7242ec4
>2012-12-17 10:08:47,953 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [470bd088] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:08:48,096 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-79) [fc992c9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:08:48,143 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5214f5ea
>2012-12-17 10:08:48,144 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 673e5de2
>2012-12-17 10:08:48,163 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:48,307 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] FINISH, SpmStopVDSCommand, log id: 673e5de2
>2012-12-17 10:08:48,337 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] FINISH, ResetIrsVDSCommand, log id: 5214f5ea
>2012-12-17 10:08:48,347 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3c3f6653
>2012-12-17 10:08:48,348 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [fc992c9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:48,348 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3c3f6653
>2012-12-17 10:08:48,394 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:08:48,406 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5065
>2012-12-17 10:08:48,407 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5065, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: a06a41b
>2012-12-17 10:08:48,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling started: taskId = 4bb6f221-dec2-4688-a27a-5d1f40b61a71
>2012-12-17 10:08:49,451 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling ended: taskId = 4bb6f221-dec2-4688-a27a-5d1f40b61a71 task status = finished
>2012-12-17 10:08:49,469 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling ended, spm status: SPM
>2012-12-17 10:08:49,471 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4bb6f221-dec2-4688-a27a-5d1f40b61a71), log id: 348837e3
>2012-12-17 10:08:49,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMClearTaskVDSCommand, log id: 348837e3
>2012-12-17 10:08:49,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5b0d3306, log id: a06a41b
>2012-12-17 10:08:49,506 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:08:49,525 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:49,527 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2fff4424
>2012-12-17 10:08:49,546 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2fff4424
>2012-12-17 10:08:49,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7242ec4
>2012-12-17 10:08:49,546 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:08:49,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1fd2b01a
>2012-12-17 10:08:49,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:49,547 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4623831a
>2012-12-17 10:08:49,566 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4623831a
>2012-12-17 10:08:49,566 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1fd2b01a
>2012-12-17 10:08:49,566 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:08:50,766 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [2b3eea84] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:08:58,440 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [786860d5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:08:58,585 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [66f997e3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:08:58,687 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [66f997e3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 646c28d4
>2012-12-17 10:08:58,689 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [66f997e3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4a397745
>2012-12-17 10:08:58,709 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [66f997e3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:58,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [66f997e3] FINISH, SpmStopVDSCommand, log id: 4a397745
>2012-12-17 10:08:58,963 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [66f997e3] FINISH, ResetIrsVDSCommand, log id: 646c28d4
>2012-12-17 10:08:58,974 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [66f997e3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 15cd5966
>2012-12-17 10:08:58,974 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [66f997e3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:08:58,974 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [66f997e3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 15cd5966
>2012-12-17 10:09:01,843 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [78864454] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:09:09,055 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:09:09,080 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5066
>2012-12-17 10:09:09,081 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5066, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7d4c679f
>2012-12-17 10:09:09,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] spmStart polling started: taskId = 2f0b7400-64b9-4c87-91a6-5a7863855b16
>2012-12-17 10:09:11,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] spmStart polling ended: taskId = 2f0b7400-64b9-4c87-91a6-5a7863855b16 task status = finished
>2012-12-17 10:09:11,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] spmStart polling ended, spm status: SPM
>2012-12-17 10:09:11,169 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2f0b7400-64b9-4c87-91a6-5a7863855b16), log id: 5a0b2d54
>2012-12-17 10:09:11,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] FINISH, HSMClearTaskVDSCommand, log id: 5a0b2d54
>2012-12-17 10:09:11,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@13cd13fb, log id: 7d4c679f
>2012-12-17 10:09:11,403 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:09:11,465 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [569d76f1] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:09:11,466 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 392366fb
>2012-12-17 10:09:11,487 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:09:11,601 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [55bb30e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:09:11,693 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 586a50a6
>2012-12-17 10:09:11,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3afc5b6f
>2012-12-17 10:09:11,715 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:09:11,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, SpmStopVDSCommand, log id: 3afc5b6f
>2012-12-17 10:09:11,885 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, ResetIrsVDSCommand, log id: 586a50a6
>2012-12-17 10:09:11,896 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2536eab8
>2012-12-17 10:09:11,896 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [55bb30e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:09:11,896 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2536eab8
>2012-12-17 10:09:11,980 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [569d76f1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:09:12,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [569d76f1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5067
>2012-12-17 10:09:12,038 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5067, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 34c30eae
>2012-12-17 10:09:12,060 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] spmStart polling started: taskId = 0b8fa183-02a7-46fd-8a77-df06fde89d1a
>2012-12-17 10:09:13,082 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] spmStart polling ended: taskId = 0b8fa183-02a7-46fd-8a77-df06fde89d1a task status = finished
>2012-12-17 10:09:13,100 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] spmStart polling ended, spm status: SPM
>2012-12-17 10:09:13,101 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0b8fa183-02a7-46fd-8a77-df06fde89d1a), log id: 4b7be55c
>2012-12-17 10:09:13,121 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, HSMClearTaskVDSCommand, log id: 4b7be55c
>2012-12-17 10:09:13,121 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@27076335, log id: 34c30eae
>2012-12-17 10:09:13,131 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [569d76f1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:09:13,144 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:09:13,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5cd2f6eb
>2012-12-17 10:09:13,171 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5cd2f6eb
>2012-12-17 10:09:13,171 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 392366fb
>2012-12-17 10:09:13,171 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [569d76f1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:09:13,171 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3cea2427
>2012-12-17 10:09:13,171 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:09:13,173 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 23f5ef78
>2012-12-17 10:09:13,192 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 23f5ef78
>2012-12-17 10:09:13,192 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3cea2427
>2012-12-17 10:09:13,192 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [569d76f1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:09:22,011 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [6827474] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:09:22,112 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [56306d9f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:09:22,181 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3f0fc259
>2012-12-17 10:09:22,182 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7b47292
>2012-12-17 10:09:22,202 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:09:22,324 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [b0e3bca] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:09:22,999 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] FINISH, SpmStopVDSCommand, log id: 7b47292
>2012-12-17 10:09:23,010 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] FINISH, ResetIrsVDSCommand, log id: 3f0fc259
>2012-12-17 10:09:23,021 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ee89afe
>2012-12-17 10:09:23,021 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [56306d9f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:09:23,021 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ee89afe
>2012-12-17 10:09:33,069 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4d407d58] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:09:33,102 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4d407d58] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5068
>2012-12-17 10:09:33,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5068, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 73a5c1dd
>2012-12-17 10:09:33,126 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] spmStart polling started: taskId = 88c02bdf-60f4-43ee-b58c-a0d5e8bcf47e
>2012-12-17 10:09:34,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] spmStart polling ended: taskId = 88c02bdf-60f4-43ee-b58c-a0d5e8bcf47e task status = finished
>2012-12-17 10:09:34,168 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] spmStart polling ended, spm status: SPM
>2012-12-17 10:09:34,169 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=88c02bdf-60f4-43ee-b58c-a0d5e8bcf47e), log id: 73a88f38
>2012-12-17 10:09:34,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] FINISH, HSMClearTaskVDSCommand, log id: 73a88f38
>2012-12-17 10:09:34,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3f52e0da, log id: 73a5c1dd
>2012-12-17 10:09:34,201 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4d407d58] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:09:34,240 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-38) [7dff3777] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:09:34,240 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 53044205
>2012-12-17 10:09:34,262 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4d407d58] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:09:34,360 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [7aeab1da] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:09:34,446 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 718b2191
>2012-12-17 10:09:34,448 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6d8a3737
>2012-12-17 10:09:34,469 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:09:35,193 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] FINISH, SpmStopVDSCommand, log id: 6d8a3737
>2012-12-17 10:09:35,215 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] FINISH, ResetIrsVDSCommand, log id: 718b2191
>2012-12-17 10:09:35,229 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3297a447
>2012-12-17 10:09:35,229 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7aeab1da] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:09:35,229 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3297a447
>2012-12-17 10:09:35,319 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [7dff3777] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:09:35,336 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [7dff3777] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5069
>2012-12-17 10:09:35,337 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5069, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6bbd9522
>2012-12-17 10:09:35,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] spmStart polling started: taskId = e7233cc1-49b9-4d5f-825c-39b7f0ee0c5b
>2012-12-17 10:09:36,383 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] spmStart polling ended: taskId = e7233cc1-49b9-4d5f-825c-39b7f0ee0c5b task status = finished
>2012-12-17 10:09:36,402 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] spmStart polling ended, spm status: SPM
>2012-12-17 10:09:36,403 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e7233cc1-49b9-4d5f-825c-39b7f0ee0c5b), log id: 3e99c483
>2012-12-17 10:09:36,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, HSMClearTaskVDSCommand, log id: 3e99c483
>2012-12-17 10:09:36,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c4d81e4, log id: 6bbd9522
>2012-12-17 10:09:36,673 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [7dff3777] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:09:36,786 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:09:36,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 75f550ca
>2012-12-17 10:09:36,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 75f550ca
>2012-12-17 10:09:36,809 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 53044205
>2012-12-17 10:09:36,809 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-38) [7dff3777] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:09:36,810 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3c8b3a46
>2012-12-17 10:09:36,810 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:09:36,811 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6f4ad571
>2012-12-17 10:09:36,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f4ad571
>2012-12-17 10:09:36,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3c8b3a46
>2012-12-17 10:09:36,830 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-38) [7dff3777] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:09:42,764 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:09:45,361 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [140420a5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:09:45,566 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [b18376] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:09:45,725 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [b18376] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5b706106
>2012-12-17 10:09:45,727 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [b18376] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 24946378
>2012-12-17 10:09:45,747 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [b18376] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:09:45,890 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [b18376] FINISH, SpmStopVDSCommand, log id: 24946378
>2012-12-17 10:09:46,273 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [b18376] FINISH, ResetIrsVDSCommand, log id: 5b706106
>2012-12-17 10:09:46,284 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [b18376] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 21df4b6e
>2012-12-17 10:09:46,284 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [b18376] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:09:46,284 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [b18376] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 21df4b6e
>2012-12-17 10:09:52,974 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:09:56,831 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [4a8b829d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:09:56,840 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [4a8b829d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5070
>2012-12-17 10:09:56,841 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5070, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 76092f7c
>2012-12-17 10:09:56,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] spmStart polling started: taskId = 4c9385b6-0830-43da-ba0b-4483c298dbd5
>2012-12-17 10:09:57,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] spmStart polling ended: taskId = 4c9385b6-0830-43da-ba0b-4483c298dbd5 task status = finished
>2012-12-17 10:09:57,904 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] spmStart polling ended, spm status: SPM
>2012-12-17 10:09:57,905 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4c9385b6-0830-43da-ba0b-4483c298dbd5), log id: 3717367e
>2012-12-17 10:09:57,925 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] FINISH, HSMClearTaskVDSCommand, log id: 3717367e
>2012-12-17 10:09:57,925 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@54faa673, log id: 76092f7c
>2012-12-17 10:09:57,941 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [4a8b829d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:09:57,994 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5c496b37] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:09:57,994 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2496725f
>2012-12-17 10:09:58,011 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [4a8b829d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:09:58,143 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:09:58,204 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2e9723c
>2012-12-17 10:09:58,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4fb38fc8
>2012-12-17 10:09:58,226 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:09:58,913 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] FINISH, SpmStopVDSCommand, log id: 4fb38fc8
>2012-12-17 10:09:58,937 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] FINISH, ResetIrsVDSCommand, log id: 2e9723c
>2012-12-17 10:09:58,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44a5a8a8
>2012-12-17 10:09:58,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:09:58,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44a5a8a8
>2012-12-17 10:09:59,001 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:09:59,015 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5071
>2012-12-17 10:09:59,016 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5071, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44e22257
>2012-12-17 10:09:59,052 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] spmStart polling started: taskId = 42a9731d-efa7-45f3-b893-ae80fa8971b1
>2012-12-17 10:10:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [29a5a9b6] Checking autorecoverable hosts
>2012-12-17 10:10:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [29a5a9b6] Autorecovering 0 hosts
>2012-12-17 10:10:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [29a5a9b6] Checking autorecoverable hosts done
>2012-12-17 10:10:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [29a5a9b6] Autorecovering storage domains is disabled, skipping
>2012-12-17 10:10:00,075 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] spmStart polling ended: taskId = 42a9731d-efa7-45f3-b893-ae80fa8971b1 task status = finished
>2012-12-17 10:10:00,093 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] spmStart polling ended, spm status: SPM
>2012-12-17 10:10:00,095 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=42a9731d-efa7-45f3-b893-ae80fa8971b1), log id: 47e964c0
>2012-12-17 10:10:00,113 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, HSMClearTaskVDSCommand, log id: 47e964c0
>2012-12-17 10:10:00,113 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2bd59e22, log id: 44e22257
>2012-12-17 10:10:00,125 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:10:00,139 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:00,143 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3f0eab15
>2012-12-17 10:10:00,163 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3f0eab15
>2012-12-17 10:10:00,164 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2496725f
>2012-12-17 10:10:00,164 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5c496b37] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:10:00,164 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: b6b2fb7
>2012-12-17 10:10:00,164 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:00,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 57292bf2
>2012-12-17 10:10:00,184 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 57292bf2
>2012-12-17 10:10:00,184 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: b6b2fb7
>2012-12-17 10:10:00,184 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5c496b37] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:10:03,220 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [5bc6d47] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:10:09,052 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:10:09,287 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-20) [5b09131c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:10:09,366 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [5b09131c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7a91f1b
>2012-12-17 10:10:09,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [5b09131c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 30ecb2eb
>2012-12-17 10:10:09,394 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [5b09131c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:09,792 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [5b09131c] FINISH, SpmStopVDSCommand, log id: 30ecb2eb
>2012-12-17 10:10:09,806 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [5b09131c] FINISH, ResetIrsVDSCommand, log id: 7a91f1b
>2012-12-17 10:10:09,817 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [5b09131c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7ae6d2cc
>2012-12-17 10:10:09,817 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [5b09131c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:09,817 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [5b09131c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7ae6d2cc
>2012-12-17 10:10:13,420 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:10:19,862 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [464dfbdd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:10:19,868 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [464dfbdd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5072
>2012-12-17 10:10:19,869 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5072, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 45d62007
>2012-12-17 10:10:19,891 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] spmStart polling started: taskId = 77ab263c-03aa-43e0-8171-7bd24b17d10a
>2012-12-17 10:10:20,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] spmStart polling ended: taskId = 77ab263c-03aa-43e0-8171-7bd24b17d10a task status = finished
>2012-12-17 10:10:20,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] spmStart polling ended, spm status: SPM
>2012-12-17 10:10:20,934 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=77ab263c-03aa-43e0-8171-7bd24b17d10a), log id: 1ed1c0c
>2012-12-17 10:10:20,953 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] FINISH, HSMClearTaskVDSCommand, log id: 1ed1c0c
>2012-12-17 10:10:20,953 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@75cc58d0, log id: 45d62007
>2012-12-17 10:10:21,482 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [464dfbdd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:10:22,106 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:10:22,106 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e87fa84
>2012-12-17 10:10:22,107 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [464dfbdd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:10:22,577 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-35) [141038b9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:10:22,678 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: b0eaaa3
>2012-12-17 10:10:22,680 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7701ea41
>2012-12-17 10:10:22,700 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [141038b9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:23,168 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, SpmStopVDSCommand, log id: 7701ea41
>2012-12-17 10:10:23,199 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, ResetIrsVDSCommand, log id: b0eaaa3
>2012-12-17 10:10:23,210 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5488eca9
>2012-12-17 10:10:23,210 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [141038b9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:23,210 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5488eca9
>2012-12-17 10:10:23,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:10:23,274 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5073
>2012-12-17 10:10:23,275 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5073, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 15e092bd
>2012-12-17 10:10:23,299 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling started: taskId = 3f4f06bc-0c36-4efd-8896-0b799e54dfe4
>2012-12-17 10:10:24,324 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended: taskId = 3f4f06bc-0c36-4efd-8896-0b799e54dfe4 task status = finished
>2012-12-17 10:10:24,342 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended, spm status: SPM
>2012-12-17 10:10:24,343 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3f4f06bc-0c36-4efd-8896-0b799e54dfe4), log id: 3aeb3bbf
>2012-12-17 10:10:24,367 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMClearTaskVDSCommand, log id: 3aeb3bbf
>2012-12-17 10:10:24,367 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6312ff79, log id: 15e092bd
>2012-12-17 10:10:24,397 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:10:24,438 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:24,442 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4d6a7cfb
>2012-12-17 10:10:24,461 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4d6a7cfb
>2012-12-17 10:10:24,461 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e87fa84
>2012-12-17 10:10:24,461 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:10:24,461 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 35916799
>2012-12-17 10:10:24,461 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:24,463 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6121e691
>2012-12-17 10:10:24,482 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6121e691
>2012-12-17 10:10:24,482 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 35916799
>2012-12-17 10:10:24,482 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:10:33,308 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [375e73d3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:10:33,400 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [15d070bf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:10:33,453 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 51d5f396
>2012-12-17 10:10:33,455 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 50033860
>2012-12-17 10:10:33,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:33,733 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, SpmStopVDSCommand, log id: 50033860
>2012-12-17 10:10:33,761 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, ResetIrsVDSCommand, log id: 51d5f396
>2012-12-17 10:10:33,771 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5cac45bc
>2012-12-17 10:10:33,771 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [15d070bf] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:33,771 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5cac45bc
>2012-12-17 10:10:33,812 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3d52876a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:10:43,846 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [6412070c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:10:43,872 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [6412070c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5074
>2012-12-17 10:10:43,874 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [6412070c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5074, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 13b98396
>2012-12-17 10:10:43,896 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [6412070c] spmStart polling started: taskId = 3753fb83-295b-4b4e-8f4a-f3bfe5146769
>2012-12-17 10:10:44,919 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [6412070c] spmStart polling ended: taskId = 3753fb83-295b-4b4e-8f4a-f3bfe5146769 task status = finished
>2012-12-17 10:10:44,938 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [6412070c] spmStart polling ended, spm status: SPM
>2012-12-17 10:10:44,939 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [6412070c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3753fb83-295b-4b4e-8f4a-f3bfe5146769), log id: 3cc01920
>2012-12-17 10:10:44,957 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [6412070c] FINISH, HSMClearTaskVDSCommand, log id: 3cc01920
>2012-12-17 10:10:44,957 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [6412070c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@67b1a308, log id: 13b98396
>2012-12-17 10:10:44,971 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [6412070c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:10:45,015 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:10:45,015 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1b424c05
>2012-12-17 10:10:45,031 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [6412070c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:10:45,158 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [102f58ae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:10:45,212 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [102f58ae] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 49aa8082
>2012-12-17 10:10:45,213 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [102f58ae] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 9d05794
>2012-12-17 10:10:45,234 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [102f58ae] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:45,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [102f58ae] FINISH, SpmStopVDSCommand, log id: 9d05794
>2012-12-17 10:10:45,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [102f58ae] FINISH, ResetIrsVDSCommand, log id: 49aa8082
>2012-12-17 10:10:45,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [102f58ae] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 38c5d7f6
>2012-12-17 10:10:45,402 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [102f58ae] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:45,402 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [102f58ae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 38c5d7f6
>2012-12-17 10:10:45,461 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:10:45,481 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5075
>2012-12-17 10:10:45,482 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5075, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2fc80f20
>2012-12-17 10:10:45,506 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling started: taskId = 46bda918-1719-43ff-949e-76e1342f143b
>2012-12-17 10:10:46,527 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling ended: taskId = 46bda918-1719-43ff-949e-76e1342f143b task status = finished
>2012-12-17 10:10:46,544 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling ended, spm status: SPM
>2012-12-17 10:10:46,546 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=46bda918-1719-43ff-949e-76e1342f143b), log id: 1a510a88
>2012-12-17 10:10:46,564 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMClearTaskVDSCommand, log id: 1a510a88
>2012-12-17 10:10:46,564 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e8a7437, log id: 2fc80f20
>2012-12-17 10:10:46,586 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:10:46,640 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:46,644 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25fb17bd
>2012-12-17 10:10:46,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25fb17bd
>2012-12-17 10:10:46,664 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1b424c05
>2012-12-17 10:10:46,664 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:10:46,664 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 51bf1f2d
>2012-12-17 10:10:46,664 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:46,666 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 56d0dca8
>2012-12-17 10:10:46,684 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 56d0dca8
>2012-12-17 10:10:46,684 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 51bf1f2d
>2012-12-17 10:10:46,684 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:10:54,378 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [2a110a6a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:10:55,506 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [2f07235e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:10:55,725 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [78bc48c3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:10:55,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4f50de86
>2012-12-17 10:10:55,813 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 452de4b9
>2012-12-17 10:10:55,833 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:56,031 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] FINISH, SpmStopVDSCommand, log id: 452de4b9
>2012-12-17 10:10:56,619 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] FINISH, ResetIrsVDSCommand, log id: 4f50de86
>2012-12-17 10:10:56,630 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1fd91ee8
>2012-12-17 10:10:56,630 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [78bc48c3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:10:56,630 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1fd91ee8
>2012-12-17 10:11:04,696 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6001c862] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:11:06,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [37e97c0a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:11:07,004 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [37e97c0a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5076
>2012-12-17 10:11:07,006 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5076, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 65848726
>2012-12-17 10:11:07,029 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] spmStart polling started: taskId = 8b61ee91-75b7-489a-a6e0-205e40c785ef
>2012-12-17 10:11:08,052 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] spmStart polling ended: taskId = 8b61ee91-75b7-489a-a6e0-205e40c785ef task status = finished
>2012-12-17 10:11:08,070 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] spmStart polling ended, spm status: SPM
>2012-12-17 10:11:08,071 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8b61ee91-75b7-489a-a6e0-205e40c785ef), log id: 7f2425b3
>2012-12-17 10:11:08,090 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] FINISH, HSMClearTaskVDSCommand, log id: 7f2425b3
>2012-12-17 10:11:08,090 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1e274bfb, log id: 65848726
>2012-12-17 10:11:08,112 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [37e97c0a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:11:08,164 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:11:08,164 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 48730772
>2012-12-17 10:11:08,174 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [37e97c0a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:11:08,329 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-97) [68ede0fd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:11:08,406 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 13ecf4a2
>2012-12-17 10:11:08,407 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5199f93d
>2012-12-17 10:11:08,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:08,562 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, SpmStopVDSCommand, log id: 5199f93d
>2012-12-17 10:11:08,573 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, ResetIrsVDSCommand, log id: 13ecf4a2
>2012-12-17 10:11:08,584 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 66941b8d
>2012-12-17 10:11:08,584 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68ede0fd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:08,584 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 66941b8d
>2012-12-17 10:11:08,656 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:11:08,677 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5077
>2012-12-17 10:11:08,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5077, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67bb3496
>2012-12-17 10:11:08,700 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling started: taskId = 6dc969dd-f65f-46dd-9ddb-3aab3c48a907
>2012-12-17 10:11:09,723 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended: taskId = 6dc969dd-f65f-46dd-9ddb-3aab3c48a907 task status = finished
>2012-12-17 10:11:09,743 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended, spm status: SPM
>2012-12-17 10:11:09,744 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6dc969dd-f65f-46dd-9ddb-3aab3c48a907), log id: 21d07285
>2012-12-17 10:11:09,762 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMClearTaskVDSCommand, log id: 21d07285
>2012-12-17 10:11:09,762 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7c0e8b1a, log id: 67bb3496
>2012-12-17 10:11:09,775 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:11:09,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:09,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4318a70
>2012-12-17 10:11:09,849 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4318a70
>2012-12-17 10:11:09,849 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 48730772
>2012-12-17 10:11:09,849 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:11:09,849 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 662de97f
>2012-12-17 10:11:09,849 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:09,850 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 600e1518
>2012-12-17 10:11:09,868 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 600e1518
>2012-12-17 10:11:09,869 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 662de97f
>2012-12-17 10:11:09,869 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:11:14,918 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:11:18,871 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [5cd90aa7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:11:19,013 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [1dd89be6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:11:19,126 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 14f2ceb4
>2012-12-17 10:11:19,128 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2e850107
>2012-12-17 10:11:19,147 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:19,739 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, SpmStopVDSCommand, log id: 2e850107
>2012-12-17 10:11:19,763 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, ResetIrsVDSCommand, log id: 14f2ceb4
>2012-12-17 10:11:19,774 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1020ace1
>2012-12-17 10:11:19,774 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:19,774 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1020ace1
>2012-12-17 10:11:25,159 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [78864454] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:11:29,851 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [2ce0ef25] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:11:29,909 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [2ce0ef25] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5078
>2012-12-17 10:11:29,910 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5078, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4c1628b
>2012-12-17 10:11:29,932 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] spmStart polling started: taskId = 38babc83-24be-4dba-a821-a8ee80d7f944
>2012-12-17 10:11:30,954 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] spmStart polling ended: taskId = 38babc83-24be-4dba-a821-a8ee80d7f944 task status = finished
>2012-12-17 10:11:30,972 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] spmStart polling ended, spm status: SPM
>2012-12-17 10:11:30,973 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=38babc83-24be-4dba-a821-a8ee80d7f944), log id: 112244d1
>2012-12-17 10:11:30,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] FINISH, HSMClearTaskVDSCommand, log id: 112244d1
>2012-12-17 10:11:30,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@60d3fe20, log id: 4c1628b
>2012-12-17 10:11:31,541 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [2ce0ef25] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:11:31,651 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [53480d94] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:11:31,652 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 39cba23e
>2012-12-17 10:11:31,698 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [2ce0ef25] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:11:31,839 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [a97381c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:11:31,912 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [a97381c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1b8b7bcb
>2012-12-17 10:11:31,913 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [a97381c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6d2d914a
>2012-12-17 10:11:31,932 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [a97381c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:32,088 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [a97381c] FINISH, SpmStopVDSCommand, log id: 6d2d914a
>2012-12-17 10:11:32,140 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [a97381c] FINISH, ResetIrsVDSCommand, log id: 1b8b7bcb
>2012-12-17 10:11:32,151 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [a97381c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 159f30a6
>2012-12-17 10:11:32,151 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [a97381c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:32,151 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [a97381c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 159f30a6
>2012-12-17 10:11:32,205 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [53480d94] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:11:32,236 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [53480d94] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5079
>2012-12-17 10:11:32,237 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5079, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6f9bae7e
>2012-12-17 10:11:32,260 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [53480d94] spmStart polling started: taskId = db7100c3-d74b-4fbe-818e-97094e5e55ef
>2012-12-17 10:11:33,282 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [53480d94] spmStart polling ended: taskId = db7100c3-d74b-4fbe-818e-97094e5e55ef task status = finished
>2012-12-17 10:11:33,300 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [53480d94] spmStart polling ended, spm status: SPM
>2012-12-17 10:11:33,301 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=db7100c3-d74b-4fbe-818e-97094e5e55ef), log id: 7e34a0ee
>2012-12-17 10:11:33,319 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, HSMClearTaskVDSCommand, log id: 7e34a0ee
>2012-12-17 10:11:33,319 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3131a9fd, log id: 6f9bae7e
>2012-12-17 10:11:33,332 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [53480d94] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:11:33,348 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:33,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 62da5df5
>2012-12-17 10:11:33,372 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 62da5df5
>2012-12-17 10:11:33,372 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 39cba23e
>2012-12-17 10:11:33,372 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [53480d94] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:11:33,372 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e4eb286
>2012-12-17 10:11:33,372 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:33,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 20ccb3f8
>2012-12-17 10:11:33,392 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 20ccb3f8
>2012-12-17 10:11:33,392 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e4eb286
>2012-12-17 10:11:33,392 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [53480d94] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:11:35,441 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [29a5a9b6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:11:42,243 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [2a110a6a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:11:42,357 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-15) [30ddd635] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:11:42,404 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3807089a
>2012-12-17 10:11:42,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5ab26649
>2012-12-17 10:11:42,425 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:42,587 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, SpmStopVDSCommand, log id: 5ab26649
>2012-12-17 10:11:42,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, ResetIrsVDSCommand, log id: 3807089a
>2012-12-17 10:11:42,682 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 253639ab
>2012-12-17 10:11:42,682 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [30ddd635] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:42,682 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 253639ab
>2012-12-17 10:11:45,661 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [78bc48c3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:11:52,750 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:11:52,766 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5080
>2012-12-17 10:11:52,767 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5080, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4a38e83e
>2012-12-17 10:11:52,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] spmStart polling started: taskId = a80cd820-a6b4-48ad-92fc-17e8eb4abca7
>2012-12-17 10:11:53,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] spmStart polling ended: taskId = a80cd820-a6b4-48ad-92fc-17e8eb4abca7 task status = finished
>2012-12-17 10:11:53,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] spmStart polling ended, spm status: SPM
>2012-12-17 10:11:53,842 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a80cd820-a6b4-48ad-92fc-17e8eb4abca7), log id: 1d6f3ece
>2012-12-17 10:11:53,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, HSMClearTaskVDSCommand, log id: 1d6f3ece
>2012-12-17 10:11:53,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7284557a, log id: 4a38e83e
>2012-12-17 10:11:53,870 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:11:53,921 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [66369950] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:11:53,921 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3bc7af
>2012-12-17 10:11:53,964 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:11:54,134 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [7e43b5c5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:11:54,198 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [7e43b5c5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 77467291
>2012-12-17 10:11:54,199 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [7e43b5c5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 521cc783
>2012-12-17 10:11:54,220 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [7e43b5c5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:54,356 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [7e43b5c5] FINISH, SpmStopVDSCommand, log id: 521cc783
>2012-12-17 10:11:54,373 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [7e43b5c5] FINISH, ResetIrsVDSCommand, log id: 77467291
>2012-12-17 10:11:54,383 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [7e43b5c5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 79b50160
>2012-12-17 10:11:54,383 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [7e43b5c5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:54,383 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [7e43b5c5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 79b50160
>2012-12-17 10:11:54,440 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [66369950] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:11:54,450 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [66369950] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5081
>2012-12-17 10:11:54,451 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5081, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1509633a
>2012-12-17 10:11:54,474 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [66369950] spmStart polling started: taskId = 3b453d1b-6459-4a81-862a-d8838f2a4bd7
>2012-12-17 10:11:55,498 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [66369950] spmStart polling ended: taskId = 3b453d1b-6459-4a81-862a-d8838f2a4bd7 task status = finished
>2012-12-17 10:11:55,521 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [66369950] spmStart polling ended, spm status: SPM
>2012-12-17 10:11:55,522 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3b453d1b-6459-4a81-862a-d8838f2a4bd7), log id: 33651cc2
>2012-12-17 10:11:55,540 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, HSMClearTaskVDSCommand, log id: 33651cc2
>2012-12-17 10:11:55,540 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@33431eea, log id: 1509633a
>2012-12-17 10:11:55,565 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [66369950] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:11:55,582 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:55,584 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3e3d2976
>2012-12-17 10:11:55,603 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3e3d2976
>2012-12-17 10:11:55,603 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3bc7af
>2012-12-17 10:11:55,603 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [66369950] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:11:55,603 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3cc942f
>2012-12-17 10:11:55,603 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:11:55,604 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4bf2bf37
>2012-12-17 10:11:55,622 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4bf2bf37
>2012-12-17 10:11:55,622 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3cc942f
>2012-12-17 10:11:55,622 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [66369950] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:11:57,345 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [141038b9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:12:04,482 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [62b3971a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:12:04,586 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [76c4364c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:12:04,664 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: d5cb2d3
>2012-12-17 10:12:04,665 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 28926125
>2012-12-17 10:12:04,687 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:12:04,855 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] FINISH, SpmStopVDSCommand, log id: 28926125
>2012-12-17 10:12:04,869 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] FINISH, ResetIrsVDSCommand, log id: d5cb2d3
>2012-12-17 10:12:04,881 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1bc2a644
>2012-12-17 10:12:04,881 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [76c4364c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:12:04,881 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1bc2a644
>2012-12-17 10:12:07,592 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:12:14,933 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:12:14,989 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5082
>2012-12-17 10:12:14,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5082, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7aa587c
>2012-12-17 10:12:15,012 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] spmStart polling started: taskId = 0e9d6095-fd53-441e-9139-0c35ffe538e5
>2012-12-17 10:12:16,033 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] spmStart polling ended: taskId = 0e9d6095-fd53-441e-9139-0c35ffe538e5 task status = finished
>2012-12-17 10:12:16,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] spmStart polling ended, spm status: SPM
>2012-12-17 10:12:16,053 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0e9d6095-fd53-441e-9139-0c35ffe538e5), log id: 43545336
>2012-12-17 10:12:16,071 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] FINISH, HSMClearTaskVDSCommand, log id: 43545336
>2012-12-17 10:12:16,071 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@552320c2, log id: 7aa587c
>2012-12-17 10:12:16,519 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:12:16,974 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [75689c2c] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:12:16,975 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 37587f02
>2012-12-17 10:12:17,060 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:12:17,213 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-34) [1630089b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:12:17,270 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [1630089b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 9edcd4d
>2012-12-17 10:12:17,272 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [1630089b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 96832f
>2012-12-17 10:12:17,292 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [1630089b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:12:17,463 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [1630089b] FINISH, SpmStopVDSCommand, log id: 96832f
>2012-12-17 10:12:17,477 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [1630089b] FINISH, ResetIrsVDSCommand, log id: 9edcd4d
>2012-12-17 10:12:17,488 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [1630089b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4384e88f
>2012-12-17 10:12:17,488 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [1630089b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:12:17,488 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [1630089b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4384e88f
>2012-12-17 10:12:17,567 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:12:17,584 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5083
>2012-12-17 10:12:17,585 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5083, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 79459cb1
>2012-12-17 10:12:17,607 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] spmStart polling started: taskId = f2e51791-d2a4-4116-8fcd-c7493ea66fc2
>2012-12-17 10:12:18,629 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] spmStart polling ended: taskId = f2e51791-d2a4-4116-8fcd-c7493ea66fc2 task status = finished
>2012-12-17 10:12:18,647 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] spmStart polling ended, spm status: SPM
>2012-12-17 10:12:18,648 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f2e51791-d2a4-4116-8fcd-c7493ea66fc2), log id: 7418afb4
>2012-12-17 10:12:18,666 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, HSMClearTaskVDSCommand, log id: 7418afb4
>2012-12-17 10:12:18,666 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@b78df8, log id: 79459cb1
>2012-12-17 10:12:18,704 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:12:18,718 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:12:18,720 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4996aeb4
>2012-12-17 10:12:18,742 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4996aeb4
>2012-12-17 10:12:18,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 37587f02
>2012-12-17 10:12:18,742 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [75689c2c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:12:18,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3e953f19
>2012-12-17 10:12:18,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:12:18,744 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 409e954c
>2012-12-17 10:12:18,762 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 409e954c
>2012-12-17 10:12:18,762 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3e953f19
>2012-12-17 10:12:18,762 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [75689c2c] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:12:27,602 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:12:27,809 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [2a51251c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:12:27,924 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6c03eee6
>2012-12-17 10:12:27,926 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 158d24c1
>2012-12-17 10:12:27,946 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:12:28,100 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] FINISH, SpmStopVDSCommand, log id: 158d24c1
>2012-12-17 10:12:28,123 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] FINISH, ResetIrsVDSCommand, log id: 6c03eee6
>2012-12-17 10:12:28,134 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a901274
>2012-12-17 10:12:28,134 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [2a51251c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:12:28,134 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a901274
>2012-12-17 10:12:28,141 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [5074b866] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:12:38,211 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3577b816] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:12:38,230 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3577b816] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5084
>2012-12-17 10:12:38,231 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [3577b816] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5084, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 346ad80a
>2012-12-17 10:12:38,256 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [3577b816] spmStart polling started: taskId = 76e5bba9-93bc-43eb-a2d5-d7860de506a5
>2012-12-17 10:12:39,278 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [3577b816] spmStart polling ended: taskId = 76e5bba9-93bc-43eb-a2d5-d7860de506a5 task status = finished
>2012-12-17 10:12:39,297 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [3577b816] spmStart polling ended, spm status: SPM
>2012-12-17 10:12:39,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [3577b816] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=76e5bba9-93bc-43eb-a2d5-d7860de506a5), log id: 77edd467
>2012-12-17 10:12:39,317 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [3577b816] FINISH, HSMClearTaskVDSCommand, log id: 77edd467
>2012-12-17 10:12:39,317 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [3577b816] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1f13f3b4, log id: 346ad80a
>2012-12-17 10:12:39,334 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3577b816] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:12:39,383 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [346a0820] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:12:39,383 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: e51a8b8
>2012-12-17 10:12:39,408 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3577b816] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:12:39,678 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [956a2f9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:12:39,751 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [956a2f9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5783844a
>2012-12-17 10:12:39,764 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [956a2f9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 224bd27f
>2012-12-17 10:12:39,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [956a2f9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:12:39,943 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [956a2f9] FINISH, SpmStopVDSCommand, log id: 224bd27f
>2012-12-17 10:12:39,958 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [956a2f9] FINISH, ResetIrsVDSCommand, log id: 5783844a
>2012-12-17 10:12:39,969 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [956a2f9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1db423fb
>2012-12-17 10:12:39,969 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [956a2f9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:12:39,969 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [956a2f9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1db423fb
>2012-12-17 10:12:40,023 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:12:40,045 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5085
>2012-12-17 10:12:40,047 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5085, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 25dfaf06
>2012-12-17 10:12:40,068 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling started: taskId = a4ec3598-0730-4372-bc48-40686c304749
>2012-12-17 10:12:41,090 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling ended: taskId = a4ec3598-0730-4372-bc48-40686c304749 task status = finished
>2012-12-17 10:12:41,109 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling ended, spm status: SPM
>2012-12-17 10:12:41,110 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a4ec3598-0730-4372-bc48-40686c304749), log id: 1c4dc490
>2012-12-17 10:12:41,130 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, HSMClearTaskVDSCommand, log id: 1c4dc490
>2012-12-17 10:12:41,130 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@35a3d874, log id: 25dfaf06
>2012-12-17 10:12:41,417 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:12:41,589 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:12:41,590 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 43d6b33f
>2012-12-17 10:12:41,610 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 43d6b33f
>2012-12-17 10:12:41,610 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: e51a8b8
>2012-12-17 10:12:41,610 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [346a0820] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:12:41,610 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 44920fe4
>2012-12-17 10:12:41,610 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:12:41,611 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 520031a6
>2012-12-17 10:12:41,629 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 520031a6
>2012-12-17 10:12:41,629 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 44920fe4
>2012-12-17 10:12:41,629 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [346a0820] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:12:48,545 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [7dff3777] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:12:50,069 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [622cf65b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:12:50,194 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [3c648024] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:12:50,242 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [3c648024] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 49183305
>2012-12-17 10:12:50,244 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [3c648024] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4795323
>2012-12-17 10:12:50,264 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [3c648024] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:12:50,414 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [3c648024] FINISH, SpmStopVDSCommand, log id: 4795323
>2012-12-17 10:12:50,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [3c648024] FINISH, ResetIrsVDSCommand, log id: 49183305
>2012-12-17 10:12:50,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [3c648024] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 22cde959
>2012-12-17 10:12:50,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [3c648024] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:12:50,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [3c648024] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 22cde959
>2012-12-17 10:12:58,782 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:13:00,486 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [40858b1a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:13:00,508 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [40858b1a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5086
>2012-12-17 10:13:00,510 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5086, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3c253d87
>2012-12-17 10:13:00,532 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] spmStart polling started: taskId = 1164a0e7-7564-46a1-bcc1-0450227fd116
>2012-12-17 10:13:01,554 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] spmStart polling ended: taskId = 1164a0e7-7564-46a1-bcc1-0450227fd116 task status = finished
>2012-12-17 10:13:01,572 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] spmStart polling ended, spm status: SPM
>2012-12-17 10:13:01,573 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1164a0e7-7564-46a1-bcc1-0450227fd116), log id: 6ccaea2c
>2012-12-17 10:13:01,591 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] FINISH, HSMClearTaskVDSCommand, log id: 6ccaea2c
>2012-12-17 10:13:01,591 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7dd42680, log id: 3c253d87
>2012-12-17 10:13:01,615 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [40858b1a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:13:01,892 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [4d8dc1c4] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:13:01,892 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 8758276
>2012-12-17 10:13:01,929 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [40858b1a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:13:02,092 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [46423891] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:13:02,230 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [46423891] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 533f5f81
>2012-12-17 10:13:02,232 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [46423891] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6c16c8ee
>2012-12-17 10:13:02,251 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [46423891] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:02,384 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [46423891] FINISH, SpmStopVDSCommand, log id: 6c16c8ee
>2012-12-17 10:13:02,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [46423891] FINISH, ResetIrsVDSCommand, log id: 533f5f81
>2012-12-17 10:13:02,410 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [46423891] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 435e6064
>2012-12-17 10:13:02,410 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [46423891] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:02,410 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [46423891] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 435e6064
>2012-12-17 10:13:02,478 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:13:02,503 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5087
>2012-12-17 10:13:02,504 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5087, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: ed55267
>2012-12-17 10:13:02,526 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] spmStart polling started: taskId = 9b634c12-e229-4665-abd4-a821b421fbc0
>2012-12-17 10:13:03,548 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] spmStart polling ended: taskId = 9b634c12-e229-4665-abd4-a821b421fbc0 task status = finished
>2012-12-17 10:13:03,566 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] spmStart polling ended, spm status: SPM
>2012-12-17 10:13:03,568 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9b634c12-e229-4665-abd4-a821b421fbc0), log id: 2b2656af
>2012-12-17 10:13:03,587 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, HSMClearTaskVDSCommand, log id: 2b2656af
>2012-12-17 10:13:03,587 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6205114a, log id: ed55267
>2012-12-17 10:13:03,598 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:13:03,665 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:03,667 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3ff8e0e5
>2012-12-17 10:13:03,687 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3ff8e0e5
>2012-12-17 10:13:03,687 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 8758276
>2012-12-17 10:13:03,687 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [4d8dc1c4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:13:03,688 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6c74fb32
>2012-12-17 10:13:03,688 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:03,689 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4c3838df
>2012-12-17 10:13:03,707 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4c3838df
>2012-12-17 10:13:03,707 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6c74fb32
>2012-12-17 10:13:03,707 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [4d8dc1c4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:13:09,089 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [53480d94] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:13:12,529 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:13:12,935 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-62) [10ed878d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:13:13,031 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [10ed878d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 44b250ed
>2012-12-17 10:13:13,032 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [10ed878d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 64d1baa2
>2012-12-17 10:13:13,052 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [10ed878d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:13,175 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [10ed878d] FINISH, SpmStopVDSCommand, log id: 64d1baa2
>2012-12-17 10:13:13,190 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [10ed878d] FINISH, ResetIrsVDSCommand, log id: 44b250ed
>2012-12-17 10:13:13,201 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [10ed878d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 501f6383
>2012-12-17 10:13:13,201 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [10ed878d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:13,201 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [10ed878d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 501f6383
>2012-12-17 10:13:19,333 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:13:23,249 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [78bc48c3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:13:23,258 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [78bc48c3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5088
>2012-12-17 10:13:23,259 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5088, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2d3573df
>2012-12-17 10:13:23,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] spmStart polling started: taskId = e990e37b-fe00-4b7c-a447-7a3953c466d5
>2012-12-17 10:13:24,302 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] spmStart polling ended: taskId = e990e37b-fe00-4b7c-a447-7a3953c466d5 task status = finished
>2012-12-17 10:13:24,320 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] spmStart polling ended, spm status: SPM
>2012-12-17 10:13:24,322 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e990e37b-fe00-4b7c-a447-7a3953c466d5), log id: 2ff10234
>2012-12-17 10:13:24,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] FINISH, HSMClearTaskVDSCommand, log id: 2ff10234
>2012-12-17 10:13:24,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@77c7bc82, log id: 2d3573df
>2012-12-17 10:13:24,382 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [78bc48c3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:13:24,462 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-22) [42fa3e4] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:13:24,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 694dc685
>2012-12-17 10:13:24,507 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [78bc48c3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:13:24,725 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [200cd224] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:13:24,778 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [200cd224] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6366f434
>2012-12-17 10:13:24,779 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [200cd224] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 535ee8c5
>2012-12-17 10:13:24,799 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [200cd224] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:24,937 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [200cd224] FINISH, SpmStopVDSCommand, log id: 535ee8c5
>2012-12-17 10:13:24,976 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [200cd224] FINISH, ResetIrsVDSCommand, log id: 6366f434
>2012-12-17 10:13:24,987 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [200cd224] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3d5aee25
>2012-12-17 10:13:24,987 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [200cd224] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:24,987 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [200cd224] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3d5aee25
>2012-12-17 10:13:25,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [42fa3e4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:13:25,073 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [42fa3e4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5089
>2012-12-17 10:13:25,075 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5089, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 32d22f5d
>2012-12-17 10:13:25,096 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] spmStart polling started: taskId = 7eb9803a-11f6-46fe-b299-3f93f6dea500
>2012-12-17 10:13:26,118 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] spmStart polling ended: taskId = 7eb9803a-11f6-46fe-b299-3f93f6dea500 task status = finished
>2012-12-17 10:13:26,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] spmStart polling ended, spm status: SPM
>2012-12-17 10:13:26,137 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7eb9803a-11f6-46fe-b299-3f93f6dea500), log id: 69cb1d1d
>2012-12-17 10:13:26,155 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, HSMClearTaskVDSCommand, log id: 69cb1d1d
>2012-12-17 10:13:26,155 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@15e4912e, log id: 32d22f5d
>2012-12-17 10:13:26,361 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [42fa3e4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:13:26,404 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:26,407 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 79c0c7e4
>2012-12-17 10:13:26,428 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 79c0c7e4
>2012-12-17 10:13:26,428 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 694dc685
>2012-12-17 10:13:26,428 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-22) [42fa3e4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:13:26,428 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5f83882e
>2012-12-17 10:13:26,428 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:26,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6eb77325
>2012-12-17 10:13:26,448 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6eb77325
>2012-12-17 10:13:26,448 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5f83882e
>2012-12-17 10:13:26,448 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-22) [42fa3e4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:13:29,681 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [406c9a80] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:13:35,120 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:13:35,361 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [46ee245a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:13:35,439 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [46ee245a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 121710b5
>2012-12-17 10:13:35,440 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [46ee245a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7a73330f
>2012-12-17 10:13:35,460 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [46ee245a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:35,618 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [46ee245a] FINISH, SpmStopVDSCommand, log id: 7a73330f
>2012-12-17 10:13:35,660 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [46ee245a] FINISH, ResetIrsVDSCommand, log id: 121710b5
>2012-12-17 10:13:35,672 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [46ee245a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 23578c64
>2012-12-17 10:13:35,672 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [46ee245a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:35,672 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [46ee245a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 23578c64
>2012-12-17 10:13:39,903 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:13:45,752 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [205b7df7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:13:45,784 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [205b7df7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5090
>2012-12-17 10:13:45,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5090, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 431717ca
>2012-12-17 10:13:45,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] spmStart polling started: taskId = 956f1e8a-e74c-4a20-bbec-ff4cb4bc2eca
>2012-12-17 10:13:46,834 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] spmStart polling ended: taskId = 956f1e8a-e74c-4a20-bbec-ff4cb4bc2eca task status = finished
>2012-12-17 10:13:46,853 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] spmStart polling ended, spm status: SPM
>2012-12-17 10:13:46,854 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=956f1e8a-e74c-4a20-bbec-ff4cb4bc2eca), log id: 5da51f60
>2012-12-17 10:13:46,872 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] FINISH, HSMClearTaskVDSCommand, log id: 5da51f60
>2012-12-17 10:13:46,873 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3af07a8, log id: 431717ca
>2012-12-17 10:13:46,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [205b7df7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:13:47,013 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:13:47,013 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 108fd21f
>2012-12-17 10:13:47,048 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [205b7df7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:13:47,224 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [5ee48710] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:13:47,286 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2b260f29
>2012-12-17 10:13:47,287 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7f0d64e1
>2012-12-17 10:13:47,307 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:47,439 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] FINISH, SpmStopVDSCommand, log id: 7f0d64e1
>2012-12-17 10:13:47,454 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] FINISH, ResetIrsVDSCommand, log id: 2b260f29
>2012-12-17 10:13:47,465 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b7849e6
>2012-12-17 10:13:47,465 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [5ee48710] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:47,465 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b7849e6
>2012-12-17 10:13:47,552 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:13:47,561 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5091
>2012-12-17 10:13:47,562 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5091, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 39d8bbb6
>2012-12-17 10:13:47,586 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling started: taskId = 7ff7293e-0037-44e9-81c5-75d3fb591e6d
>2012-12-17 10:13:48,607 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling ended: taskId = 7ff7293e-0037-44e9-81c5-75d3fb591e6d task status = finished
>2012-12-17 10:13:48,625 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling ended, spm status: SPM
>2012-12-17 10:13:48,626 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7ff7293e-0037-44e9-81c5-75d3fb591e6d), log id: 6fad1af8
>2012-12-17 10:13:48,649 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMClearTaskVDSCommand, log id: 6fad1af8
>2012-12-17 10:13:48,649 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@274233c1, log id: 39d8bbb6
>2012-12-17 10:13:48,677 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:13:48,697 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:48,698 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 31d3e2ba
>2012-12-17 10:13:48,718 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31d3e2ba
>2012-12-17 10:13:48,718 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 108fd21f
>2012-12-17 10:13:48,718 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:13:48,718 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 17e076a2
>2012-12-17 10:13:48,718 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:48,719 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 516903d8
>2012-12-17 10:13:48,738 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 516903d8
>2012-12-17 10:13:48,738 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 17e076a2
>2012-12-17 10:13:48,738 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:13:50,136 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:13:57,614 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:13:57,909 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [1bc6b95c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:13:57,967 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [1bc6b95c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7fcab695
>2012-12-17 10:13:57,969 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [1bc6b95c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2552b646
>2012-12-17 10:13:57,988 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [1bc6b95c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:58,170 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [1bc6b95c] FINISH, SpmStopVDSCommand, log id: 2552b646
>2012-12-17 10:13:58,186 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [1bc6b95c] FINISH, ResetIrsVDSCommand, log id: 7fcab695
>2012-12-17 10:13:58,197 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [1bc6b95c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2a474e66
>2012-12-17 10:13:58,197 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [1bc6b95c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:13:58,197 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [1bc6b95c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2a474e66
>2012-12-17 10:14:00,342 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [10ed878d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:14:08,265 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [56306d9f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:14:08,304 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [56306d9f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5092
>2012-12-17 10:14:08,305 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5092, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 872c360
>2012-12-17 10:14:08,328 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] spmStart polling started: taskId = a963af9b-1799-41b8-b1d1-bf414ee80b70
>2012-12-17 10:14:09,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] spmStart polling ended: taskId = a963af9b-1799-41b8-b1d1-bf414ee80b70 task status = finished
>2012-12-17 10:14:09,366 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] spmStart polling ended, spm status: SPM
>2012-12-17 10:14:09,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a963af9b-1799-41b8-b1d1-bf414ee80b70), log id: 12dd1aa7
>2012-12-17 10:14:09,385 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] FINISH, HSMClearTaskVDSCommand, log id: 12dd1aa7
>2012-12-17 10:14:09,385 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7200d39d, log id: 872c360
>2012-12-17 10:14:09,395 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [56306d9f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:14:09,438 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [26a9278b] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:14:09,438 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1cc55c4e
>2012-12-17 10:14:09,447 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [56306d9f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:14:09,540 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [2747eada] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:14:09,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [2747eada] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 44d418dd
>2012-12-17 10:14:09,602 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [2747eada] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 331fb3ae
>2012-12-17 10:14:09,622 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [2747eada] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:09,763 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [2747eada] FINISH, SpmStopVDSCommand, log id: 331fb3ae
>2012-12-17 10:14:09,778 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [2747eada] FINISH, ResetIrsVDSCommand, log id: 44d418dd
>2012-12-17 10:14:09,788 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [2747eada] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5bca3556
>2012-12-17 10:14:09,788 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [2747eada] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:09,788 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [2747eada] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5bca3556
>2012-12-17 10:14:09,848 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [26a9278b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:14:09,864 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [26a9278b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5093
>2012-12-17 10:14:09,865 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5093, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4e04ebd5
>2012-12-17 10:14:09,887 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] spmStart polling started: taskId = 60012190-4cec-4788-ab27-3d523720d397
>2012-12-17 10:14:10,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] spmStart polling ended: taskId = 60012190-4cec-4788-ab27-3d523720d397 task status = finished
>2012-12-17 10:14:10,926 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] spmStart polling ended, spm status: SPM
>2012-12-17 10:14:10,927 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=60012190-4cec-4788-ab27-3d523720d397), log id: 3531d15d
>2012-12-17 10:14:10,945 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, HSMClearTaskVDSCommand, log id: 3531d15d
>2012-12-17 10:14:10,945 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@59a84bf3, log id: 4e04ebd5
>2012-12-17 10:14:11,336 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [26a9278b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:14:11,370 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:11,374 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 43f13f49
>2012-12-17 10:14:11,394 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 43f13f49
>2012-12-17 10:14:11,394 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1cc55c4e
>2012-12-17 10:14:11,394 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [26a9278b] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:14:11,394 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1e68fc41
>2012-12-17 10:14:11,394 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:11,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 350304dc
>2012-12-17 10:14:11,413 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 350304dc
>2012-12-17 10:14:11,413 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1e68fc41
>2012-12-17 10:14:11,413 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [26a9278b] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:14:19,907 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [66369950] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:14:20,012 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [4020b36c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:14:20,081 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [4020b36c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2a61f606
>2012-12-17 10:14:20,082 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [4020b36c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7d750389
>2012-12-17 10:14:20,102 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [4020b36c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:20,239 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [4020b36c] FINISH, SpmStopVDSCommand, log id: 7d750389
>2012-12-17 10:14:20,323 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [4020b36c] FINISH, ResetIrsVDSCommand, log id: 2a61f606
>2012-12-17 10:14:20,335 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [4020b36c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 58d0e627
>2012-12-17 10:14:20,335 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [4020b36c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:20,335 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [4020b36c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 58d0e627
>2012-12-17 10:14:20,827 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [fc992c9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:14:30,401 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:14:30,435 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5094
>2012-12-17 10:14:30,437 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5094, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4861a6f6
>2012-12-17 10:14:30,460 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling started: taskId = 20541ac5-775c-466d-8db7-7b29dcec6be9
>2012-12-17 10:14:31,485 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling ended: taskId = 20541ac5-775c-466d-8db7-7b29dcec6be9 task status = finished
>2012-12-17 10:14:31,503 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling ended, spm status: SPM
>2012-12-17 10:14:31,504 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=20541ac5-775c-466d-8db7-7b29dcec6be9), log id: 24bf9497
>2012-12-17 10:14:31,523 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, HSMClearTaskVDSCommand, log id: 24bf9497
>2012-12-17 10:14:31,523 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1db4585a, log id: 4861a6f6
>2012-12-17 10:14:31,809 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:14:31,872 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [15d070bf] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:14:31,873 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2a5a2463
>2012-12-17 10:14:31,906 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:14:32,089 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [2fe721f0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:14:32,183 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [2fe721f0] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 43e1f14b
>2012-12-17 10:14:32,184 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [2fe721f0] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5a9d6ffe
>2012-12-17 10:14:32,205 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [2fe721f0] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:32,357 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [2fe721f0] FINISH, SpmStopVDSCommand, log id: 5a9d6ffe
>2012-12-17 10:14:32,387 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [2fe721f0] FINISH, ResetIrsVDSCommand, log id: 43e1f14b
>2012-12-17 10:14:32,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [2fe721f0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3125a2da
>2012-12-17 10:14:32,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [2fe721f0] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:32,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [2fe721f0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3125a2da
>2012-12-17 10:14:32,458 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [15d070bf] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:14:32,471 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [15d070bf] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5095
>2012-12-17 10:14:32,472 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5095, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5de5da1f
>2012-12-17 10:14:32,495 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] spmStart polling started: taskId = 928e028b-4b17-40ff-933b-de1a5181d1b5
>2012-12-17 10:14:33,518 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] spmStart polling ended: taskId = 928e028b-4b17-40ff-933b-de1a5181d1b5 task status = finished
>2012-12-17 10:14:33,536 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] spmStart polling ended, spm status: SPM
>2012-12-17 10:14:33,537 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=928e028b-4b17-40ff-933b-de1a5181d1b5), log id: 7a310499
>2012-12-17 10:14:33,556 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, HSMClearTaskVDSCommand, log id: 7a310499
>2012-12-17 10:14:33,556 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2825177b, log id: 5de5da1f
>2012-12-17 10:14:33,586 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [15d070bf] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:14:33,618 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:33,622 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1d668a3d
>2012-12-17 10:14:33,646 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1d668a3d
>2012-12-17 10:14:33,646 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2a5a2463
>2012-12-17 10:14:33,646 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [15d070bf] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:14:33,646 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: a439aba
>2012-12-17 10:14:33,646 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:33,647 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 112e9bdf
>2012-12-17 10:14:33,666 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 112e9bdf
>2012-12-17 10:14:33,666 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a439aba
>2012-12-17 10:14:33,666 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [15d070bf] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:14:42,093 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:14:42,511 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [a97381c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:14:42,733 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [78c0579f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:14:42,862 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [78c0579f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4fd9d3c
>2012-12-17 10:14:42,863 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [78c0579f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1e87adb0
>2012-12-17 10:14:42,883 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [78c0579f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:43,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [78c0579f] FINISH, SpmStopVDSCommand, log id: 1e87adb0
>2012-12-17 10:14:43,086 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [78c0579f] FINISH, ResetIrsVDSCommand, log id: 4fd9d3c
>2012-12-17 10:14:43,097 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [78c0579f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 50f6f322
>2012-12-17 10:14:43,097 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [78c0579f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:43,097 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [78c0579f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 50f6f322
>2012-12-17 10:14:52,352 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [18826afe] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:14:53,173 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [2a51251c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:14:53,185 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [2a51251c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5096
>2012-12-17 10:14:53,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5096, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5787bc95
>2012-12-17 10:14:53,210 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] spmStart polling started: taskId = 262970bc-a8af-4216-9ac4-3701896876ff
>2012-12-17 10:14:54,232 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] spmStart polling ended: taskId = 262970bc-a8af-4216-9ac4-3701896876ff task status = finished
>2012-12-17 10:14:54,251 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] spmStart polling ended, spm status: SPM
>2012-12-17 10:14:54,252 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=262970bc-a8af-4216-9ac4-3701896876ff), log id: 7eda0cc9
>2012-12-17 10:14:54,270 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] FINISH, HSMClearTaskVDSCommand, log id: 7eda0cc9
>2012-12-17 10:14:54,270 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1d7da12f, log id: 5787bc95
>2012-12-17 10:14:54,296 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [2a51251c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:14:54,328 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [536ede7] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:14:54,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 370468db
>2012-12-17 10:14:54,373 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [2a51251c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:14:54,593 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [533115ff] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:14:54,697 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [533115ff] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4e9b044a
>2012-12-17 10:14:54,699 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [533115ff] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2db47c5f
>2012-12-17 10:14:54,718 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [533115ff] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:54,866 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [533115ff] FINISH, SpmStopVDSCommand, log id: 2db47c5f
>2012-12-17 10:14:54,882 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [533115ff] FINISH, ResetIrsVDSCommand, log id: 4e9b044a
>2012-12-17 10:14:54,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [533115ff] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 467665d1
>2012-12-17 10:14:54,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [533115ff] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:54,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [533115ff] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 467665d1
>2012-12-17 10:14:54,944 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [536ede7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:14:54,956 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [536ede7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5097
>2012-12-17 10:14:54,957 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5097, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1ddcacd6
>2012-12-17 10:14:54,979 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] spmStart polling started: taskId = 02348d43-bd2a-41d1-bbcb-da5b007cd10b
>2012-12-17 10:14:56,009 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] spmStart polling ended: taskId = 02348d43-bd2a-41d1-bbcb-da5b007cd10b task status = finished
>2012-12-17 10:14:56,028 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] spmStart polling ended, spm status: SPM
>2012-12-17 10:14:56,030 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=02348d43-bd2a-41d1-bbcb-da5b007cd10b), log id: 25eef2d4
>2012-12-17 10:14:56,048 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, HSMClearTaskVDSCommand, log id: 25eef2d4
>2012-12-17 10:14:56,048 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a884fc0, log id: 1ddcacd6
>2012-12-17 10:14:56,447 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [536ede7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:14:56,486 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:56,490 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 78347a2
>2012-12-17 10:14:56,510 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 78347a2
>2012-12-17 10:14:56,510 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 370468db
>2012-12-17 10:14:56,510 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [536ede7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:14:56,510 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3bd3e088
>2012-12-17 10:14:56,510 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:14:56,512 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5b04c78e
>2012-12-17 10:14:56,531 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5b04c78e
>2012-12-17 10:14:56,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3bd3e088
>2012-12-17 10:14:56,531 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [536ede7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:15:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [4e8c7167] Checking autorecoverable hosts
>2012-12-17 10:15:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [4e8c7167] Autorecovering 0 hosts
>2012-12-17 10:15:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [4e8c7167] Checking autorecoverable hosts done
>2012-12-17 10:15:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [4e8c7167] Autorecovering storage domains is disabled, skipping
>2012-12-17 10:15:02,554 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [515f648d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:15:04,974 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [3ced8e0c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:15:05,056 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [2d5b99a4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:15:05,104 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 46ea23f4
>2012-12-17 10:15:05,106 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 60c8e2ac
>2012-12-17 10:15:05,125 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:15:05,294 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] FINISH, SpmStopVDSCommand, log id: 60c8e2ac
>2012-12-17 10:15:05,309 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] FINISH, ResetIrsVDSCommand, log id: 46ea23f4
>2012-12-17 10:15:05,320 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c7f0b09
>2012-12-17 10:15:05,321 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2d5b99a4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:15:05,321 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c7f0b09
>2012-12-17 10:15:13,141 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:15:15,383 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7c8941c5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:15:15,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7c8941c5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5098
>2012-12-17 10:15:15,411 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5098, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5283430
>2012-12-17 10:15:15,434 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] spmStart polling started: taskId = 4a4b71df-6f27-4e1e-88d5-48b1973c1b69
>2012-12-17 10:15:16,457 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] spmStart polling ended: taskId = 4a4b71df-6f27-4e1e-88d5-48b1973c1b69 task status = finished
>2012-12-17 10:15:16,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] spmStart polling ended, spm status: SPM
>2012-12-17 10:15:16,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4a4b71df-6f27-4e1e-88d5-48b1973c1b69), log id: 36a2d1a3
>2012-12-17 10:15:16,494 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, HSMClearTaskVDSCommand, log id: 36a2d1a3
>2012-12-17 10:15:16,495 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7b71405a, log id: 5283430
>2012-12-17 10:15:16,788 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7c8941c5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:15:17,073 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-51) [56fd87e7] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:15:17,073 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 13c0d10d
>2012-12-17 10:15:17,124 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7c8941c5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:15:17,331 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [6e88e48] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:15:17,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [6e88e48] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1432b1
>2012-12-17 10:15:17,411 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [6e88e48] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f1cc073
>2012-12-17 10:15:17,431 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [6e88e48] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:15:17,567 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [6e88e48] FINISH, SpmStopVDSCommand, log id: 4f1cc073
>2012-12-17 10:15:17,588 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [6e88e48] FINISH, ResetIrsVDSCommand, log id: 1432b1
>2012-12-17 10:15:17,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [6e88e48] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 17677baa
>2012-12-17 10:15:17,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [6e88e48] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:15:17,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [6e88e48] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 17677baa
>2012-12-17 10:15:17,658 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:15:17,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5099
>2012-12-17 10:15:17,673 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5099, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 68912f85
>2012-12-17 10:15:17,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] spmStart polling started: taskId = dc7ea60f-0932-45b3-9c1b-554cb22b5560
>2012-12-17 10:15:18,717 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] spmStart polling ended: taskId = dc7ea60f-0932-45b3-9c1b-554cb22b5560 task status = finished
>2012-12-17 10:15:18,735 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] spmStart polling ended, spm status: SPM
>2012-12-17 10:15:18,736 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dc7ea60f-0932-45b3-9c1b-554cb22b5560), log id: 70730870
>2012-12-17 10:15:18,755 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, HSMClearTaskVDSCommand, log id: 70730870
>2012-12-17 10:15:18,755 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@102f2abd, log id: 68912f85
>2012-12-17 10:15:18,779 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:15:18,810 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:15:18,814 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2c993a7d
>2012-12-17 10:15:18,833 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2c993a7d
>2012-12-17 10:15:18,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 13c0d10d
>2012-12-17 10:15:18,833 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-51) [56fd87e7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:15:18,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7fcefb6a
>2012-12-17 10:15:18,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:15:18,835 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3444cdf7
>2012-12-17 10:15:18,853 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3444cdf7
>2012-12-17 10:15:18,853 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7fcefb6a
>2012-12-17 10:15:18,853 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-51) [56fd87e7] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:15:23,340 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [6c52cc2e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:15:27,699 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [39a6c2d3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:15:27,957 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-58) [5cdf4906] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:15:28,025 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [5cdf4906] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 45f1ed
>2012-12-17 10:15:28,026 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [5cdf4906] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5e4623d2
>2012-12-17 10:15:28,046 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [5cdf4906] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:15:28,223 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [5cdf4906] FINISH, SpmStopVDSCommand, log id: 5e4623d2
>2012-12-17 10:15:28,234 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [5cdf4906] FINISH, ResetIrsVDSCommand, log id: 45f1ed
>2012-12-17 10:15:28,244 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [5cdf4906] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3215811c
>2012-12-17 10:15:28,244 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [5cdf4906] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:15:28,245 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [5cdf4906] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3215811c
>2012-12-17 10:15:33,555 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [569d76f1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:15:38,313 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:15:38,326 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5100
>2012-12-17 10:15:38,328 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5100, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: a20c5c1
>2012-12-17 10:15:38,349 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] spmStart polling started: taskId = 6f06bcc6-a5b7-418a-8d78-539da367b99c
>2012-12-17 10:15:39,371 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] spmStart polling ended: taskId = 6f06bcc6-a5b7-418a-8d78-539da367b99c task status = finished
>2012-12-17 10:15:39,390 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] spmStart polling ended, spm status: SPM
>2012-12-17 10:15:39,392 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6f06bcc6-a5b7-418a-8d78-539da367b99c), log id: 410b5d00
>2012-12-17 10:15:39,410 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] FINISH, HSMClearTaskVDSCommand, log id: 410b5d00
>2012-12-17 10:15:39,411 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5106cc5c, log id: a20c5c1
>2012-12-17 10:15:39,439 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:15:39,498 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [30ddd635] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:15:39,498 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3a411948
>2012-12-17 10:15:39,510 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:15:39,718 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [32d3383b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:15:39,797 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [32d3383b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 595a5542
>2012-12-17 10:15:39,798 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [32d3383b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6495b2c
>2012-12-17 10:15:39,819 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [32d3383b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:15:39,946 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [32d3383b] FINISH, SpmStopVDSCommand, log id: 6495b2c
>2012-12-17 10:15:39,973 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [32d3383b] FINISH, ResetIrsVDSCommand, log id: 595a5542
>2012-12-17 10:15:39,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [32d3383b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 461c963c
>2012-12-17 10:15:39,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [32d3383b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:15:39,984 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [32d3383b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 461c963c
>2012-12-17 10:15:40,069 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [30ddd635] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:15:40,090 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [30ddd635] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5101
>2012-12-17 10:15:40,092 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5101, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 29ddcd3c
>2012-12-17 10:15:40,114 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] spmStart polling started: taskId = 8762ecbd-ac43-4da8-9d1f-7adc03ba7589
>2012-12-17 10:15:41,137 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] spmStart polling ended: taskId = 8762ecbd-ac43-4da8-9d1f-7adc03ba7589 task status = finished
>2012-12-17 10:15:41,156 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] spmStart polling ended, spm status: SPM
>2012-12-17 10:15:41,157 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8762ecbd-ac43-4da8-9d1f-7adc03ba7589), log id: 576a851e
>2012-12-17 10:15:41,175 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, HSMClearTaskVDSCommand, log id: 576a851e
>2012-12-17 10:15:41,175 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1249a168, log id: 29ddcd3c
>2012-12-17 10:15:41,331 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [30ddd635] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:15:41,594 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:15:41,596 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4cb77a3f
>2012-12-17 10:15:41,617 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4cb77a3f
>2012-12-17 10:15:41,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3a411948
>2012-12-17 10:15:41,617 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [30ddd635] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:15:41,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 21bae572
>2012-12-17 10:15:41,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:15:41,618 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 617505aa
>2012-12-17 10:15:41,637 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 617505aa
>2012-12-17 10:15:41,637 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 21bae572
>2012-12-17 10:15:41,637 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [30ddd635] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:15:43,752 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:15:50,107 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:15:50,237 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-49) [5c1b0caf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:15:50,369 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 687b4483
>2012-12-17 10:15:50,371 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1e97a3b1
>2012-12-17 10:15:50,391 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:15:50,554 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, SpmStopVDSCommand, log id: 1e97a3b1
>2012-12-17 10:15:50,569 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, ResetIrsVDSCommand, log id: 687b4483
>2012-12-17 10:15:50,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7deaa4e2
>2012-12-17 10:15:50,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [5c1b0caf] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:15:50,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7deaa4e2
>2012-12-17 10:15:54,066 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:16:00,637 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [3c648024] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:16:00,703 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [3c648024] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5102
>2012-12-17 10:16:00,705 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3c648024] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5102, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7f5fb8ca
>2012-12-17 10:16:00,727 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3c648024] spmStart polling started: taskId = b7eee7cb-c8eb-44b1-ae06-cc8c5c41dea5
>2012-12-17 10:16:01,750 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3c648024] spmStart polling ended: taskId = b7eee7cb-c8eb-44b1-ae06-cc8c5c41dea5 task status = finished
>2012-12-17 10:16:01,768 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3c648024] spmStart polling ended, spm status: SPM
>2012-12-17 10:16:01,769 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [3c648024] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b7eee7cb-c8eb-44b1-ae06-cc8c5c41dea5), log id: 2a72dbe3
>2012-12-17 10:16:01,788 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [3c648024] FINISH, HSMClearTaskVDSCommand, log id: 2a72dbe3
>2012-12-17 10:16:01,788 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3c648024] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5ac229da, log id: 7f5fb8ca
>2012-12-17 10:16:01,795 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [3c648024] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:16:01,840 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [26d9e89a] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:16:01,840 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5219c606
>2012-12-17 10:16:01,862 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [3c648024] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:16:01,958 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [69c9c5a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:16:02,007 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [69c9c5a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2db609e5
>2012-12-17 10:16:02,008 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [69c9c5a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: c0501bc
>2012-12-17 10:16:02,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [69c9c5a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:02,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [69c9c5a] FINISH, SpmStopVDSCommand, log id: c0501bc
>2012-12-17 10:16:02,210 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [69c9c5a] FINISH, ResetIrsVDSCommand, log id: 2db609e5
>2012-12-17 10:16:02,221 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [69c9c5a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 79ae179c
>2012-12-17 10:16:02,221 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [69c9c5a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:02,221 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [69c9c5a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 79ae179c
>2012-12-17 10:16:02,275 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:16:02,288 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5103
>2012-12-17 10:16:02,289 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5103, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4ca586c4
>2012-12-17 10:16:02,311 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling started: taskId = 2b8934c8-b5d0-4e48-b529-a2f155fb83db
>2012-12-17 10:16:03,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling ended: taskId = 2b8934c8-b5d0-4e48-b529-a2f155fb83db task status = finished
>2012-12-17 10:16:03,359 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling ended, spm status: SPM
>2012-12-17 10:16:03,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2b8934c8-b5d0-4e48-b529-a2f155fb83db), log id: 7b33ef9
>2012-12-17 10:16:03,380 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, HSMClearTaskVDSCommand, log id: 7b33ef9
>2012-12-17 10:16:03,380 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@8eb5162, log id: 4ca586c4
>2012-12-17 10:16:03,404 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:16:03,424 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:03,426 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 31195e2f
>2012-12-17 10:16:03,444 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31195e2f
>2012-12-17 10:16:03,444 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5219c606
>2012-12-17 10:16:03,444 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [26d9e89a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:16:03,445 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 787678d7
>2012-12-17 10:16:03,445 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:03,446 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 72dfdbda
>2012-12-17 10:16:03,465 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 72dfdbda
>2012-12-17 10:16:03,465 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 787678d7
>2012-12-17 10:16:03,465 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [26d9e89a] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:16:04,302 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:16:12,314 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [46423891] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:16:12,449 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [79d25c3b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:16:12,503 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2a201b58
>2012-12-17 10:16:12,504 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: da7943d
>2012-12-17 10:16:12,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:13,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, SpmStopVDSCommand, log id: da7943d
>2012-12-17 10:16:13,047 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, ResetIrsVDSCommand, log id: 2a201b58
>2012-12-17 10:16:13,058 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3ba0d7dd
>2012-12-17 10:16:13,058 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [79d25c3b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:13,058 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3ba0d7dd
>2012-12-17 10:16:14,577 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:16:23,154 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [29a5a9b6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:16:23,176 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [29a5a9b6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5104
>2012-12-17 10:16:23,177 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5104, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3f6700c2
>2012-12-17 10:16:23,201 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] spmStart polling started: taskId = d8d4633d-4ea2-4d9d-bd43-950974be2c30
>2012-12-17 10:16:24,224 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] spmStart polling ended: taskId = d8d4633d-4ea2-4d9d-bd43-950974be2c30 task status = finished
>2012-12-17 10:16:24,244 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] spmStart polling ended, spm status: SPM
>2012-12-17 10:16:24,245 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d8d4633d-4ea2-4d9d-bd43-950974be2c30), log id: 430ab48
>2012-12-17 10:16:24,263 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] FINISH, HSMClearTaskVDSCommand, log id: 430ab48
>2012-12-17 10:16:24,263 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@173d061e, log id: 3f6700c2
>2012-12-17 10:16:24,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [29a5a9b6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:16:24,334 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-19) [3ad075f5] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:16:24,335 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 69efb109
>2012-12-17 10:16:24,358 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [29a5a9b6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:16:24,648 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-80) [13905503] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:16:24,709 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [13905503] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 16e82304
>2012-12-17 10:16:24,710 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [13905503] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5a48c4d1
>2012-12-17 10:16:24,737 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [13905503] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:24,825 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:16:24,886 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [13905503] FINISH, SpmStopVDSCommand, log id: 5a48c4d1
>2012-12-17 10:16:24,899 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [13905503] FINISH, ResetIrsVDSCommand, log id: 16e82304
>2012-12-17 10:16:24,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [13905503] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6384100c
>2012-12-17 10:16:24,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [13905503] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:24,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [13905503] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6384100c
>2012-12-17 10:16:24,981 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3ad075f5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:16:25,012 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3ad075f5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5105
>2012-12-17 10:16:25,014 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5105, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6f5eb76e
>2012-12-17 10:16:25,036 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] spmStart polling started: taskId = 8ca9c49b-7e4b-4019-96e7-f2fb114437e9
>2012-12-17 10:16:26,059 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] spmStart polling ended: taskId = 8ca9c49b-7e4b-4019-96e7-f2fb114437e9 task status = finished
>2012-12-17 10:16:26,077 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] spmStart polling ended, spm status: SPM
>2012-12-17 10:16:26,079 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8ca9c49b-7e4b-4019-96e7-f2fb114437e9), log id: 2483dfea
>2012-12-17 10:16:26,098 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, HSMClearTaskVDSCommand, log id: 2483dfea
>2012-12-17 10:16:26,098 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4de18e60, log id: 6f5eb76e
>2012-12-17 10:16:26,670 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3ad075f5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:16:26,847 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:26,850 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4d90002a
>2012-12-17 10:16:26,879 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4d90002a
>2012-12-17 10:16:26,879 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 69efb109
>2012-12-17 10:16:26,879 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-19) [3ad075f5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:16:26,879 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 28faed2e
>2012-12-17 10:16:26,880 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:26,881 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 520f8da0
>2012-12-17 10:16:26,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 520f8da0
>2012-12-17 10:16:26,900 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 28faed2e
>2012-12-17 10:16:26,900 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-19) [3ad075f5] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:16:26,940 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [b0e3bca] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:16:26,940 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3545dd1e
>2012-12-17 10:16:26,940 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:26,942 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1f3e6e23
>2012-12-17 10:16:26,960 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1f3e6e23
>2012-12-17 10:16:26,960 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3545dd1e
>2012-12-17 10:16:26,960 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [b0e3bca] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:16:26,961 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3bb0cbdb
>2012-12-17 10:16:26,961 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:26,962 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 77eb9fff
>2012-12-17 10:16:26,981 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 77eb9fff
>2012-12-17 10:16:26,981 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3bb0cbdb
>2012-12-17 10:16:26,981 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [b0e3bca] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:16:35,076 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [6038c0b1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:16:35,233 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-18) [3cb6b88f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:16:35,327 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5ec3305b
>2012-12-17 10:16:35,329 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6a8d0aca
>2012-12-17 10:16:35,348 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:35,521 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, SpmStopVDSCommand, log id: 6a8d0aca
>2012-12-17 10:16:35,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, ResetIrsVDSCommand, log id: 5ec3305b
>2012-12-17 10:16:35,557 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ed5db70
>2012-12-17 10:16:35,557 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [3cb6b88f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:35,557 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ed5db70
>2012-12-17 10:16:45,314 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [141038b9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:16:45,627 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [40e7f581] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:16:45,666 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [40e7f581] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5106
>2012-12-17 10:16:45,667 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5106, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 739b3cb6
>2012-12-17 10:16:45,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] spmStart polling started: taskId = b5e1cda6-0ff2-4850-87e1-9465105daad2
>2012-12-17 10:16:46,763 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] spmStart polling ended: taskId = b5e1cda6-0ff2-4850-87e1-9465105daad2 task status = finished
>2012-12-17 10:16:46,781 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] spmStart polling ended, spm status: SPM
>2012-12-17 10:16:46,782 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b5e1cda6-0ff2-4850-87e1-9465105daad2), log id: 74af36e1
>2012-12-17 10:16:46,800 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, HSMClearTaskVDSCommand, log id: 74af36e1
>2012-12-17 10:16:46,800 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1b3c29a0, log id: 739b3cb6
>2012-12-17 10:16:46,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [40e7f581] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:16:46,849 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:16:46,849 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 36812675
>2012-12-17 10:16:46,893 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [40e7f581] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:16:47,053 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [127fd955] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:16:47,112 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [127fd955] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 66bb91f1
>2012-12-17 10:16:47,113 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [127fd955] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5af5c697
>2012-12-17 10:16:47,133 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [127fd955] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:47,248 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [127fd955] FINISH, SpmStopVDSCommand, log id: 5af5c697
>2012-12-17 10:16:47,264 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [127fd955] FINISH, ResetIrsVDSCommand, log id: 66bb91f1
>2012-12-17 10:16:47,274 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [127fd955] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 744a1f62
>2012-12-17 10:16:47,274 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [127fd955] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:47,274 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [127fd955] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 744a1f62
>2012-12-17 10:16:47,322 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:16:47,332 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5107
>2012-12-17 10:16:47,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5107, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 36a656ae
>2012-12-17 10:16:47,365 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling started: taskId = c606391d-35de-4551-9519-993dd589a644
>2012-12-17 10:16:48,387 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended: taskId = c606391d-35de-4551-9519-993dd589a644 task status = finished
>2012-12-17 10:16:48,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended, spm status: SPM
>2012-12-17 10:16:48,406 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c606391d-35de-4551-9519-993dd589a644), log id: 655480d
>2012-12-17 10:16:48,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMClearTaskVDSCommand, log id: 655480d
>2012-12-17 10:16:48,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48ed50ba, log id: 36a656ae
>2012-12-17 10:16:48,440 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:16:48,471 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:48,473 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 71a206a7
>2012-12-17 10:16:48,494 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 71a206a7
>2012-12-17 10:16:48,494 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 36812675
>2012-12-17 10:16:48,494 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:16:48,494 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3d968942
>2012-12-17 10:16:48,494 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:48,496 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4b85a9e9
>2012-12-17 10:16:48,514 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b85a9e9
>2012-12-17 10:16:48,514 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3d968942
>2012-12-17 10:16:48,514 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:16:55,599 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:16:57,363 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [78864454] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:16:57,479 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-68) [454eb5a9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:16:57,541 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [454eb5a9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 569acbba
>2012-12-17 10:16:57,542 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [454eb5a9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2bbe0d29
>2012-12-17 10:16:57,562 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [454eb5a9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:57,767 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [454eb5a9] FINISH, SpmStopVDSCommand, log id: 2bbe0d29
>2012-12-17 10:16:57,782 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [454eb5a9] FINISH, ResetIrsVDSCommand, log id: 569acbba
>2012-12-17 10:16:57,793 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [454eb5a9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6888cc13
>2012-12-17 10:16:57,793 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [454eb5a9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:16:57,793 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [454eb5a9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6888cc13
>2012-12-17 10:17:05,813 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [102f58ae] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:17:07,875 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [a355c3f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:17:07,884 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [a355c3f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5108
>2012-12-17 10:17:07,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5108, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 395a16a0
>2012-12-17 10:17:07,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] spmStart polling started: taskId = 589c6560-0846-4894-bdb7-7cd929d3c2c1
>2012-12-17 10:17:08,943 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] spmStart polling ended: taskId = 589c6560-0846-4894-bdb7-7cd929d3c2c1 task status = finished
>2012-12-17 10:17:08,961 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] spmStart polling ended, spm status: SPM
>2012-12-17 10:17:08,963 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=589c6560-0846-4894-bdb7-7cd929d3c2c1), log id: 173d7d1c
>2012-12-17 10:17:08,981 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] FINISH, HSMClearTaskVDSCommand, log id: 173d7d1c
>2012-12-17 10:17:08,981 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@45e0c52f, log id: 395a16a0
>2012-12-17 10:17:09,013 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [a355c3f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:17:09,045 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5c496b37] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:17:09,046 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: d6176a5
>2012-12-17 10:17:09,074 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [a355c3f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:17:09,209 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [57fa1ec] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:17:09,297 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [57fa1ec] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4488b9a8
>2012-12-17 10:17:09,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [57fa1ec] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 41dbaf69
>2012-12-17 10:17:09,318 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [57fa1ec] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:09,457 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [57fa1ec] FINISH, SpmStopVDSCommand, log id: 41dbaf69
>2012-12-17 10:17:09,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [57fa1ec] FINISH, ResetIrsVDSCommand, log id: 4488b9a8
>2012-12-17 10:17:09,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [57fa1ec] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24b3e4f1
>2012-12-17 10:17:09,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [57fa1ec] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:09,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [57fa1ec] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24b3e4f1
>2012-12-17 10:17:09,560 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:17:09,573 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5109
>2012-12-17 10:17:09,574 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5109, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3dfcc9ee
>2012-12-17 10:17:09,596 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] spmStart polling started: taskId = 7ba80757-2150-4dbe-93d3-4028a1c17260
>2012-12-17 10:17:10,618 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] spmStart polling ended: taskId = 7ba80757-2150-4dbe-93d3-4028a1c17260 task status = finished
>2012-12-17 10:17:10,637 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] spmStart polling ended, spm status: SPM
>2012-12-17 10:17:10,638 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7ba80757-2150-4dbe-93d3-4028a1c17260), log id: 3a286dd
>2012-12-17 10:17:10,657 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, HSMClearTaskVDSCommand, log id: 3a286dd
>2012-12-17 10:17:10,657 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@23441908, log id: 3dfcc9ee
>2012-12-17 10:17:10,664 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:17:10,678 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:10,679 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 60f10208
>2012-12-17 10:17:10,699 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 60f10208
>2012-12-17 10:17:10,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: d6176a5
>2012-12-17 10:17:10,699 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5c496b37] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:17:10,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6ebc8cf0
>2012-12-17 10:17:10,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:10,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3931a871
>2012-12-17 10:17:10,720 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3931a871
>2012-12-17 10:17:10,720 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6ebc8cf0
>2012-12-17 10:17:10,720 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5c496b37] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:17:16,892 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:17:19,600 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [2747eada] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:17:19,708 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [780047c3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:17:19,850 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [780047c3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4097a227
>2012-12-17 10:17:19,852 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [780047c3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 359a8573
>2012-12-17 10:17:19,871 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [780047c3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:19,984 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [780047c3] FINISH, SpmStopVDSCommand, log id: 359a8573
>2012-12-17 10:17:20,021 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [780047c3] FINISH, ResetIrsVDSCommand, log id: 4097a227
>2012-12-17 10:17:20,032 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [780047c3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e797fda
>2012-12-17 10:17:20,032 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [780047c3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:20,032 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [780047c3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e797fda
>2012-12-17 10:17:27,126 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:17:30,096 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [fc992c9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:17:30,131 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [fc992c9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5110
>2012-12-17 10:17:30,132 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5110, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 22a9f199
>2012-12-17 10:17:30,153 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] spmStart polling started: taskId = e878e899-6a23-489b-bd97-546667c6aba6
>2012-12-17 10:17:31,192 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] spmStart polling ended: taskId = e878e899-6a23-489b-bd97-546667c6aba6 task status = finished
>2012-12-17 10:17:31,210 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] spmStart polling ended, spm status: SPM
>2012-12-17 10:17:31,211 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e878e899-6a23-489b-bd97-546667c6aba6), log id: 4b21677f
>2012-12-17 10:17:31,229 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] FINISH, HSMClearTaskVDSCommand, log id: 4b21677f
>2012-12-17 10:17:31,229 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7e11e961, log id: 22a9f199
>2012-12-17 10:17:31,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [fc992c9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:17:31,880 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [141038b9] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:17:31,880 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 49f5f158
>2012-12-17 10:17:31,895 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [fc992c9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:17:32,072 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-79) [938c7ef] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:17:32,206 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [938c7ef] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 75a27495
>2012-12-17 10:17:32,208 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [938c7ef] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5f71cc96
>2012-12-17 10:17:32,230 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [938c7ef] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:32,376 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [938c7ef] FINISH, SpmStopVDSCommand, log id: 5f71cc96
>2012-12-17 10:17:32,407 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [938c7ef] FINISH, ResetIrsVDSCommand, log id: 75a27495
>2012-12-17 10:17:32,418 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [938c7ef] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: e4bce3
>2012-12-17 10:17:32,418 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [938c7ef] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:32,418 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [938c7ef] FINISH, MarkPoolInReconstructModeVDSCommand, log id: e4bce3
>2012-12-17 10:17:32,472 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [141038b9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:17:32,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [141038b9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5111
>2012-12-17 10:17:32,485 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5111, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1fdf7b37
>2012-12-17 10:17:32,507 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [141038b9] spmStart polling started: taskId = f1f1d1e0-bfaf-4f83-85d7-8dc8a8447855
>2012-12-17 10:17:33,529 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [141038b9] spmStart polling ended: taskId = f1f1d1e0-bfaf-4f83-85d7-8dc8a8447855 task status = finished
>2012-12-17 10:17:33,548 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [141038b9] spmStart polling ended, spm status: SPM
>2012-12-17 10:17:33,549 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f1f1d1e0-bfaf-4f83-85d7-8dc8a8447855), log id: 7e48ee5d
>2012-12-17 10:17:33,567 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, HSMClearTaskVDSCommand, log id: 7e48ee5d
>2012-12-17 10:17:33,568 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@66195c70, log id: 1fdf7b37
>2012-12-17 10:17:33,575 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [141038b9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:17:33,598 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:33,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 63f97ca1
>2012-12-17 10:17:33,622 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 63f97ca1
>2012-12-17 10:17:33,622 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 49f5f158
>2012-12-17 10:17:33,622 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [141038b9] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:17:33,622 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6fd7504a
>2012-12-17 10:17:33,622 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:33,623 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4cced4a1
>2012-12-17 10:17:33,642 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4cced4a1
>2012-12-17 10:17:33,642 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6fd7504a
>2012-12-17 10:17:33,642 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [141038b9] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:17:37,439 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [7dff3777] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:17:42,535 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [2fe721f0] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:17:42,699 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [17997e65] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:17:42,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [17997e65] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17a1bcad
>2012-12-17 10:17:42,746 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [17997e65] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: e415e67
>2012-12-17 10:17:42,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [17997e65] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:42,947 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [17997e65] FINISH, SpmStopVDSCommand, log id: e415e67
>2012-12-17 10:17:42,960 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [17997e65] FINISH, ResetIrsVDSCommand, log id: 17a1bcad
>2012-12-17 10:17:42,971 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [17997e65] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 71e1794e
>2012-12-17 10:17:42,972 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [17997e65] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:42,972 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [17997e65] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 71e1794e
>2012-12-17 10:17:47,659 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:17:53,016 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:17:53,028 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5112
>2012-12-17 10:17:53,029 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5112, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 73cf0ae0
>2012-12-17 10:17:53,050 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] spmStart polling started: taskId = 1df34274-a1cd-41d5-a943-acb5d65200aa
>2012-12-17 10:17:54,071 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] spmStart polling ended: taskId = 1df34274-a1cd-41d5-a943-acb5d65200aa task status = finished
>2012-12-17 10:17:54,089 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] spmStart polling ended, spm status: SPM
>2012-12-17 10:17:54,090 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1df34274-a1cd-41d5-a943-acb5d65200aa), log id: 7090aaf
>2012-12-17 10:17:54,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, HSMClearTaskVDSCommand, log id: 7090aaf
>2012-12-17 10:17:54,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@629c6ac4, log id: 73cf0ae0
>2012-12-17 10:17:54,126 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:17:54,193 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [55bb30e] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:17:54,193 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4fc34169
>2012-12-17 10:17:54,251 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:17:54,437 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-54) [44e35e2b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:17:54,498 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [44e35e2b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: fe9b236
>2012-12-17 10:17:54,499 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [44e35e2b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 9152f53
>2012-12-17 10:17:54,519 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [44e35e2b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:54,682 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [44e35e2b] FINISH, SpmStopVDSCommand, log id: 9152f53
>2012-12-17 10:17:54,691 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [44e35e2b] FINISH, ResetIrsVDSCommand, log id: fe9b236
>2012-12-17 10:17:54,702 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [44e35e2b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a656cc9
>2012-12-17 10:17:54,703 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [44e35e2b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:54,703 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [44e35e2b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a656cc9
>2012-12-17 10:17:54,815 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [55bb30e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:17:54,826 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [55bb30e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5113
>2012-12-17 10:17:54,827 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5113, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: e3373b9
>2012-12-17 10:17:54,850 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] spmStart polling started: taskId = bd72d14a-bbc3-42f0-9ca0-9f7a6e10e4c6
>2012-12-17 10:17:55,871 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] spmStart polling ended: taskId = bd72d14a-bbc3-42f0-9ca0-9f7a6e10e4c6 task status = finished
>2012-12-17 10:17:55,890 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] spmStart polling ended, spm status: SPM
>2012-12-17 10:17:55,891 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bd72d14a-bbc3-42f0-9ca0-9f7a6e10e4c6), log id: 5ce31ea5
>2012-12-17 10:17:55,909 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, HSMClearTaskVDSCommand, log id: 5ce31ea5
>2012-12-17 10:17:55,909 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42f11247, log id: e3373b9
>2012-12-17 10:17:56,175 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [55bb30e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:17:56,202 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:56,203 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 63a12f6e
>2012-12-17 10:17:56,235 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 63a12f6e
>2012-12-17 10:17:56,235 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4fc34169
>2012-12-17 10:17:56,235 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [55bb30e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:17:56,235 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6b3111db
>2012-12-17 10:17:56,235 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:17:56,237 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7e24ea83
>2012-12-17 10:17:56,266 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7e24ea83
>2012-12-17 10:17:56,266 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6b3111db
>2012-12-17 10:17:56,266 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [55bb30e] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:17:57,877 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [3da48235] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:18:04,856 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:18:04,945 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [e1a392] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:18:04,995 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [e1a392] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 532823bc
>2012-12-17 10:18:04,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [e1a392] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1e673eb5
>2012-12-17 10:18:05,017 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [e1a392] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:18:05,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [e1a392] FINISH, SpmStopVDSCommand, log id: 1e673eb5
>2012-12-17 10:18:05,192 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [e1a392] FINISH, ResetIrsVDSCommand, log id: 532823bc
>2012-12-17 10:18:05,204 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [e1a392] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 573408d6
>2012-12-17 10:18:05,204 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [e1a392] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:18:05,204 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [e1a392] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 573408d6
>2012-12-17 10:18:08,144 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [780047c3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:18:15,262 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:18:15,299 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5114
>2012-12-17 10:18:15,301 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5114, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 55625bea
>2012-12-17 10:18:15,323 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling started: taskId = 7e24c51d-1a32-4a8e-bdc4-9c32d6e04b81
>2012-12-17 10:18:16,344 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling ended: taskId = 7e24c51d-1a32-4a8e-bdc4-9c32d6e04b81 task status = finished
>2012-12-17 10:18:16,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling ended, spm status: SPM
>2012-12-17 10:18:16,363 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7e24c51d-1a32-4a8e-bdc4-9c32d6e04b81), log id: 793c7cd0
>2012-12-17 10:18:16,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, HSMClearTaskVDSCommand, log id: 793c7cd0
>2012-12-17 10:18:16,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@722679de, log id: 55625bea
>2012-12-17 10:18:16,455 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:18:16,907 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [68ede0fd] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:18:16,908 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 73bbcf2c
>2012-12-17 10:18:16,930 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:18:17,176 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-40) [6431f17a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:18:17,244 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-40) [6431f17a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 666a6f3d
>2012-12-17 10:18:17,245 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-40) [6431f17a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3bb9e08
>2012-12-17 10:18:17,265 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-40) [6431f17a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:18:17,402 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-40) [6431f17a] FINISH, SpmStopVDSCommand, log id: 3bb9e08
>2012-12-17 10:18:17,415 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-40) [6431f17a] FINISH, ResetIrsVDSCommand, log id: 666a6f3d
>2012-12-17 10:18:17,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-40) [6431f17a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 48c9e822
>2012-12-17 10:18:17,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [6431f17a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:18:17,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-40) [6431f17a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 48c9e822
>2012-12-17 10:18:17,474 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68ede0fd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:18:17,488 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68ede0fd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5115
>2012-12-17 10:18:17,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5115, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6cfdf2ec
>2012-12-17 10:18:17,527 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] spmStart polling started: taskId = 73c8c1ff-a6a3-4be3-95ca-ead5f566d244
>2012-12-17 10:18:18,548 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] spmStart polling ended: taskId = 73c8c1ff-a6a3-4be3-95ca-ead5f566d244 task status = finished
>2012-12-17 10:18:18,567 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] spmStart polling ended, spm status: SPM
>2012-12-17 10:18:18,568 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=73c8c1ff-a6a3-4be3-95ca-ead5f566d244), log id: 5dc40eeb
>2012-12-17 10:18:18,586 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, HSMClearTaskVDSCommand, log id: 5dc40eeb
>2012-12-17 10:18:18,586 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@72af9664, log id: 6cfdf2ec
>2012-12-17 10:18:18,611 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68ede0fd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:18:18,626 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:18:18,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 31731b61
>2012-12-17 10:18:18,646 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31731b61
>2012-12-17 10:18:18,646 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 73bbcf2c
>2012-12-17 10:18:18,646 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [68ede0fd] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:18:18,647 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1b27e6f4
>2012-12-17 10:18:18,647 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:18:18,648 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5182656e
>2012-12-17 10:18:18,667 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5182656e
>2012-12-17 10:18:18,667 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1b27e6f4
>2012-12-17 10:18:18,667 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [68ede0fd] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:18:27,515 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [2420ee59] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:18:27,630 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [25e2aebf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:18:27,689 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [25e2aebf] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1d98384
>2012-12-17 10:18:27,691 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [25e2aebf] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 316cd540
>2012-12-17 10:18:27,712 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [25e2aebf] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:18:27,907 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [25e2aebf] FINISH, SpmStopVDSCommand, log id: 316cd540
>2012-12-17 10:18:27,932 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [25e2aebf] FINISH, ResetIrsVDSCommand, log id: 1d98384
>2012-12-17 10:18:27,943 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [25e2aebf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: bd470ec
>2012-12-17 10:18:27,943 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [25e2aebf] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:18:27,943 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [25e2aebf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: bd470ec
>2012-12-17 10:18:28,533 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:18:37,997 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:18:38,013 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5116
>2012-12-17 10:18:38,015 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5116, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1bcea1c9
>2012-12-17 10:18:38,049 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling started: taskId = 7cb350bc-a383-4d21-a5a1-117b0ed6c656
>2012-12-17 10:18:39,069 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended: taskId = 7cb350bc-a383-4d21-a5a1-117b0ed6c656 task status = finished
>2012-12-17 10:18:39,088 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended, spm status: SPM
>2012-12-17 10:18:39,089 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7cb350bc-a383-4d21-a5a1-117b0ed6c656), log id: 3c4ae40f
>2012-12-17 10:18:39,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMClearTaskVDSCommand, log id: 3c4ae40f
>2012-12-17 10:18:39,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@58c2b3b5, log id: 1bcea1c9
>2012-12-17 10:18:39,121 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:18:39,174 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [7646beaa] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:18:39,174 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7f0dca0a
>2012-12-17 10:18:39,185 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:18:39,377 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-88) [3c8f0aba] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:18:39,513 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-88) [3c8f0aba] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: eaa26d6
>2012-12-17 10:18:39,514 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [3c8f0aba] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2e3a94b9
>2012-12-17 10:18:39,534 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [3c8f0aba] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:18:39,754 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [3c8f0aba] FINISH, SpmStopVDSCommand, log id: 2e3a94b9
>2012-12-17 10:18:39,769 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-88) [3c8f0aba] FINISH, ResetIrsVDSCommand, log id: eaa26d6
>2012-12-17 10:18:39,780 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [3c8f0aba] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 28c3a303
>2012-12-17 10:18:39,780 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [3c8f0aba] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:18:39,780 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [3c8f0aba] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 28c3a303
>2012-12-17 10:18:39,837 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:18:39,858 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5117
>2012-12-17 10:18:39,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5117, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 31580d36
>2012-12-17 10:18:39,882 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] spmStart polling started: taskId = 1b16cf0e-80d0-4f4a-bfa2-8076804ce07d
>2012-12-17 10:18:40,903 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] spmStart polling ended: taskId = 1b16cf0e-80d0-4f4a-bfa2-8076804ce07d task status = finished
>2012-12-17 10:18:40,921 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] spmStart polling ended, spm status: SPM
>2012-12-17 10:18:40,923 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1b16cf0e-80d0-4f4a-bfa2-8076804ce07d), log id: 395a8525
>2012-12-17 10:18:40,941 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, HSMClearTaskVDSCommand, log id: 395a8525
>2012-12-17 10:18:40,941 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@65db5d8d, log id: 31580d36
>2012-12-17 10:18:41,682 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:18:41,769 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:18:41,773 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1ba23bd9
>2012-12-17 10:18:41,793 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ba23bd9
>2012-12-17 10:18:41,793 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7f0dca0a
>2012-12-17 10:18:41,793 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [7646beaa] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:18:41,793 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 380731bb
>2012-12-17 10:18:41,793 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:18:41,794 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 109eab13
>2012-12-17 10:18:41,813 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 109eab13
>2012-12-17 10:18:41,813 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 380731bb
>2012-12-17 10:18:41,813 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [7646beaa] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:18:48,994 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [18826afe] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:18:49,938 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3ad075f5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:18:50,080 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [4fc51f8f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:18:50,160 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [4fc51f8f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 140d8ac6
>2012-12-17 10:18:50,162 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [4fc51f8f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1fc90ff4
>2012-12-17 10:18:50,182 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [4fc51f8f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:18:50,337 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [4fc51f8f] FINISH, SpmStopVDSCommand, log id: 1fc90ff4
>2012-12-17 10:18:50,366 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [4fc51f8f] FINISH, ResetIrsVDSCommand, log id: 140d8ac6
>2012-12-17 10:18:50,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [4fc51f8f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3940f4ae
>2012-12-17 10:18:50,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [4fc51f8f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:18:50,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [4fc51f8f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3940f4ae
>2012-12-17 10:18:59,225 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [4e8c7167] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:19:00,457 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6001c862] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:19:00,466 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6001c862] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5118
>2012-12-17 10:19:00,468 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6001c862] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5118, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7f894981
>2012-12-17 10:19:00,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6001c862] spmStart polling started: taskId = e804ced3-95f5-47f9-81fc-31b0cb50506c
>2012-12-17 10:19:01,511 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6001c862] spmStart polling ended: taskId = e804ced3-95f5-47f9-81fc-31b0cb50506c task status = finished
>2012-12-17 10:19:01,529 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6001c862] spmStart polling ended, spm status: SPM
>2012-12-17 10:19:01,530 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6001c862] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e804ced3-95f5-47f9-81fc-31b0cb50506c), log id: 12e718a3
>2012-12-17 10:19:01,549 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6001c862] FINISH, HSMClearTaskVDSCommand, log id: 12e718a3
>2012-12-17 10:19:01,549 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6001c862] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2027f07, log id: 7f894981
>2012-12-17 10:19:01,562 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6001c862] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:19:01,614 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [5c1b0caf] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:19:01,614 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 9d42aec
>2012-12-17 10:19:01,639 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6001c862] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:19:01,741 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-100) [681720a4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:19:01,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [681720a4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 25de93d
>2012-12-17 10:19:01,834 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [681720a4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 75eed51f
>2012-12-17 10:19:01,854 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [681720a4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:02,036 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [681720a4] FINISH, SpmStopVDSCommand, log id: 75eed51f
>2012-12-17 10:19:02,051 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [681720a4] FINISH, ResetIrsVDSCommand, log id: 25de93d
>2012-12-17 10:19:02,062 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [681720a4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b788166
>2012-12-17 10:19:02,062 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [681720a4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:02,062 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [681720a4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b788166
>2012-12-17 10:19:02,142 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [5c1b0caf] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:19:02,164 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [5c1b0caf] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5119
>2012-12-17 10:19:02,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5119, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 720928b6
>2012-12-17 10:19:02,188 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] spmStart polling started: taskId = 3341a462-6fb3-4ab2-ae1f-fc266c74d36e
>2012-12-17 10:19:03,210 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] spmStart polling ended: taskId = 3341a462-6fb3-4ab2-ae1f-fc266c74d36e task status = finished
>2012-12-17 10:19:03,229 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] spmStart polling ended, spm status: SPM
>2012-12-17 10:19:03,231 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3341a462-6fb3-4ab2-ae1f-fc266c74d36e), log id: 25553b5d
>2012-12-17 10:19:03,249 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, HSMClearTaskVDSCommand, log id: 25553b5d
>2012-12-17 10:19:03,249 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7eac69d0, log id: 720928b6
>2012-12-17 10:19:03,273 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [5c1b0caf] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:19:03,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:03,295 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 59863e4d
>2012-12-17 10:19:03,315 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 59863e4d
>2012-12-17 10:19:03,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 9d42aec
>2012-12-17 10:19:03,315 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [5c1b0caf] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:19:03,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 21a2b5f0
>2012-12-17 10:19:03,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:03,317 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6109e86a
>2012-12-17 10:19:03,335 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6109e86a
>2012-12-17 10:19:03,335 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 21a2b5f0
>2012-12-17 10:19:03,335 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [5c1b0caf] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:19:09,510 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [141038b9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:19:12,174 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [69c9c5a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:19:12,306 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [36ca77a1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:19:12,375 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [36ca77a1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3552fc3e
>2012-12-17 10:19:12,377 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [36ca77a1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 10c2dea6
>2012-12-17 10:19:12,397 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [36ca77a1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:12,581 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [36ca77a1] FINISH, SpmStopVDSCommand, log id: 10c2dea6
>2012-12-17 10:19:12,632 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [36ca77a1] FINISH, ResetIrsVDSCommand, log id: 3552fc3e
>2012-12-17 10:19:12,643 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [36ca77a1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: b20adec
>2012-12-17 10:19:12,643 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [36ca77a1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:12,643 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [36ca77a1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: b20adec
>2012-12-17 10:19:19,951 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [15d070bf] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:19:22,791 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [1db2b76] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:19:22,854 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [1db2b76] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5120
>2012-12-17 10:19:22,856 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5120, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 79199a47
>2012-12-17 10:19:22,880 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] spmStart polling started: taskId = 49ad612e-458d-4192-aeab-bdad0c7dfc72
>2012-12-17 10:19:23,904 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] spmStart polling ended: taskId = 49ad612e-458d-4192-aeab-bdad0c7dfc72 task status = finished
>2012-12-17 10:19:23,922 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] spmStart polling ended, spm status: SPM
>2012-12-17 10:19:23,923 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=49ad612e-458d-4192-aeab-bdad0c7dfc72), log id: 4091e65d
>2012-12-17 10:19:23,942 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] FINISH, HSMClearTaskVDSCommand, log id: 4091e65d
>2012-12-17 10:19:23,942 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2aab04c9, log id: 79199a47
>2012-12-17 10:19:23,953 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [1db2b76] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:19:23,983 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:19:23,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2c5709e6
>2012-12-17 10:19:24,035 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [1db2b76] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:19:24,215 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [50119c91] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:19:24,292 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [50119c91] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5bdd0bfe
>2012-12-17 10:19:24,294 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [50119c91] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 24f8f070
>2012-12-17 10:19:24,315 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [50119c91] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:24,455 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [50119c91] FINISH, SpmStopVDSCommand, log id: 24f8f070
>2012-12-17 10:19:24,505 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [50119c91] FINISH, ResetIrsVDSCommand, log id: 5bdd0bfe
>2012-12-17 10:19:24,517 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [50119c91] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a0afb40
>2012-12-17 10:19:24,517 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [50119c91] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:24,517 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [50119c91] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a0afb40
>2012-12-17 10:19:24,564 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:19:24,589 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5121
>2012-12-17 10:19:24,590 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5121, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4eea7534
>2012-12-17 10:19:24,614 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling started: taskId = 4ad99371-bfc7-4f9a-9433-4505cf7c3ec7
>2012-12-17 10:19:25,636 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling ended: taskId = 4ad99371-bfc7-4f9a-9433-4505cf7c3ec7 task status = finished
>2012-12-17 10:19:25,655 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling ended, spm status: SPM
>2012-12-17 10:19:25,657 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4ad99371-bfc7-4f9a-9433-4505cf7c3ec7), log id: 5665b3b7
>2012-12-17 10:19:25,675 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMClearTaskVDSCommand, log id: 5665b3b7
>2012-12-17 10:19:25,675 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2ef9dfb1, log id: 4eea7534
>2012-12-17 10:19:25,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:19:25,719 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:25,721 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 8d99e05
>2012-12-17 10:19:25,741 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 8d99e05
>2012-12-17 10:19:25,741 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2c5709e6
>2012-12-17 10:19:25,741 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:19:25,741 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7155ecd
>2012-12-17 10:19:25,741 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:25,742 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 61c7b000
>2012-12-17 10:19:25,762 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 61c7b000
>2012-12-17 10:19:25,762 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7155ecd
>2012-12-17 10:19:25,762 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:19:30,194 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [569d76f1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:19:34,654 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:19:34,866 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [60aea922] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:19:34,970 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6fb341b0
>2012-12-17 10:19:34,972 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4c906331
>2012-12-17 10:19:34,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [60aea922] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:35,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, SpmStopVDSCommand, log id: 4c906331
>2012-12-17 10:19:35,204 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, ResetIrsVDSCommand, log id: 6fb341b0
>2012-12-17 10:19:35,215 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 65a9a53e
>2012-12-17 10:19:35,215 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [60aea922] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:35,215 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 65a9a53e
>2012-12-17 10:19:40,413 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [533115ff] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:19:45,295 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [200cd224] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:19:45,341 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [200cd224] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5122
>2012-12-17 10:19:45,342 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [200cd224] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5122, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44709dd2
>2012-12-17 10:19:45,366 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [200cd224] spmStart polling started: taskId = d0dc1c79-b50a-42a3-a66a-ebcd90b5f359
>2012-12-17 10:19:46,389 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [200cd224] spmStart polling ended: taskId = d0dc1c79-b50a-42a3-a66a-ebcd90b5f359 task status = finished
>2012-12-17 10:19:46,407 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [200cd224] spmStart polling ended, spm status: SPM
>2012-12-17 10:19:46,408 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [200cd224] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d0dc1c79-b50a-42a3-a66a-ebcd90b5f359), log id: 115ec9b7
>2012-12-17 10:19:46,431 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [200cd224] FINISH, HSMClearTaskVDSCommand, log id: 115ec9b7
>2012-12-17 10:19:46,431 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [200cd224] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@46ee53f1, log id: 44709dd2
>2012-12-17 10:19:46,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [200cd224] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:19:46,710 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [3cb6b88f] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:19:46,710 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 610e35df
>2012-12-17 10:19:46,728 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [200cd224] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:19:46,876 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [8102db] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:19:46,943 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [8102db] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 53799e9f
>2012-12-17 10:19:46,944 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [8102db] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7618654b
>2012-12-17 10:19:46,965 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [8102db] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:47,099 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [8102db] FINISH, SpmStopVDSCommand, log id: 7618654b
>2012-12-17 10:19:47,111 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [8102db] FINISH, ResetIrsVDSCommand, log id: 53799e9f
>2012-12-17 10:19:47,123 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [8102db] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4abf0385
>2012-12-17 10:19:47,123 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [8102db] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:47,123 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [8102db] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4abf0385
>2012-12-17 10:19:47,199 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [3cb6b88f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:19:47,213 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [3cb6b88f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5123
>2012-12-17 10:19:47,214 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5123, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3609d864
>2012-12-17 10:19:47,236 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] spmStart polling started: taskId = d7728829-bb32-4334-befa-ab2fcd3c50d4
>2012-12-17 10:19:48,259 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] spmStart polling ended: taskId = d7728829-bb32-4334-befa-ab2fcd3c50d4 task status = finished
>2012-12-17 10:19:48,278 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] spmStart polling ended, spm status: SPM
>2012-12-17 10:19:48,280 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d7728829-bb32-4334-befa-ab2fcd3c50d4), log id: 53e85008
>2012-12-17 10:19:48,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, HSMClearTaskVDSCommand, log id: 53e85008
>2012-12-17 10:19:48,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@38de6841, log id: 3609d864
>2012-12-17 10:19:48,318 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [3cb6b88f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:19:48,329 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:48,331 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 65a4eea4
>2012-12-17 10:19:48,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 65a4eea4
>2012-12-17 10:19:48,353 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 610e35df
>2012-12-17 10:19:48,353 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [3cb6b88f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:19:48,353 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 58cc8cc5
>2012-12-17 10:19:48,353 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:48,355 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7bbd3c10
>2012-12-17 10:19:48,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7bbd3c10
>2012-12-17 10:19:48,373 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 58cc8cc5
>2012-12-17 10:19:48,373 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [3cb6b88f] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:19:50,586 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:19:57,253 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:19:57,386 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-95) [75377a97] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:19:57,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-95) [75377a97] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1dbc60e8
>2012-12-17 10:19:57,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [75377a97] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 51a4121a
>2012-12-17 10:19:57,448 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [75377a97] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:57,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [75377a97] FINISH, SpmStopVDSCommand, log id: 51a4121a
>2012-12-17 10:19:57,644 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-95) [75377a97] FINISH, ResetIrsVDSCommand, log id: 1dbc60e8
>2012-12-17 10:19:57,655 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [75377a97] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24265102
>2012-12-17 10:19:57,655 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [75377a97] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:19:57,655 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [75377a97] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24265102
>2012-12-17 10:20:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [7dff3777] Checking autorecoverable hosts
>2012-12-17 10:20:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [7dff3777] Autorecovering 0 hosts
>2012-12-17 10:20:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [7dff3777] Checking autorecoverable hosts done
>2012-12-17 10:20:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [7dff3777] Autorecovering storage domains is disabled, skipping
>2012-12-17 10:20:00,795 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:20:07,705 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:20:07,745 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5124
>2012-12-17 10:20:07,747 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5124, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5985ce4e
>2012-12-17 10:20:07,770 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] spmStart polling started: taskId = 38e12a43-ffd9-4120-a2b9-848c60396943
>2012-12-17 10:20:08,792 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] spmStart polling ended: taskId = 38e12a43-ffd9-4120-a2b9-848c60396943 task status = finished
>2012-12-17 10:20:08,810 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] spmStart polling ended, spm status: SPM
>2012-12-17 10:20:08,812 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=38e12a43-ffd9-4120-a2b9-848c60396943), log id: a1c0486
>2012-12-17 10:20:08,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, HSMClearTaskVDSCommand, log id: a1c0486
>2012-12-17 10:20:08,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@356cfe99, log id: 5985ce4e
>2012-12-17 10:20:08,865 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:20:08,908 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [79d25c3b] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:20:08,908 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6932924c
>2012-12-17 10:20:08,949 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:20:09,135 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [2c0da98a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:20:09,209 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [2c0da98a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 472c10f8
>2012-12-17 10:20:09,210 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [2c0da98a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3645e52d
>2012-12-17 10:20:09,230 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [2c0da98a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:09,399 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [2c0da98a] FINISH, SpmStopVDSCommand, log id: 3645e52d
>2012-12-17 10:20:09,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [2c0da98a] FINISH, ResetIrsVDSCommand, log id: 472c10f8
>2012-12-17 10:20:09,420 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [2c0da98a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 34eea7e4
>2012-12-17 10:20:09,420 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [2c0da98a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:09,420 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [2c0da98a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 34eea7e4
>2012-12-17 10:20:09,471 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [79d25c3b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:20:09,491 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [79d25c3b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5125
>2012-12-17 10:20:09,493 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5125, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6f95c1e
>2012-12-17 10:20:09,516 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] spmStart polling started: taskId = f9e9ee3a-af98-4826-ac5a-5949472d36e9
>2012-12-17 10:20:10,539 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] spmStart polling ended: taskId = f9e9ee3a-af98-4826-ac5a-5949472d36e9 task status = finished
>2012-12-17 10:20:10,558 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] spmStart polling ended, spm status: SPM
>2012-12-17 10:20:10,560 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f9e9ee3a-af98-4826-ac5a-5949472d36e9), log id: 770cbb73
>2012-12-17 10:20:10,578 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, HSMClearTaskVDSCommand, log id: 770cbb73
>2012-12-17 10:20:10,578 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@64e6b814, log id: 6f95c1e
>2012-12-17 10:20:10,589 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [79d25c3b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:20:10,616 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:10,620 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: af38e67
>2012-12-17 10:20:10,639 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: af38e67
>2012-12-17 10:20:10,639 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6932924c
>2012-12-17 10:20:10,639 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [79d25c3b] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:20:10,640 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 79172b00
>2012-12-17 10:20:10,640 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:10,641 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 713b2576
>2012-12-17 10:20:10,659 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 713b2576
>2012-12-17 10:20:10,659 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 79172b00
>2012-12-17 10:20:10,659 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [79d25c3b] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:20:11,427 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:20:19,503 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [1bc6b95c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:20:19,653 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [65d7a0e6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:20:19,707 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 218a5785
>2012-12-17 10:20:19,708 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2311b0dd
>2012-12-17 10:20:19,729 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:19,891 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, SpmStopVDSCommand, log id: 2311b0dd
>2012-12-17 10:20:19,902 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, ResetIrsVDSCommand, log id: 218a5785
>2012-12-17 10:20:19,913 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 12b71069
>2012-12-17 10:20:19,913 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [65d7a0e6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:19,913 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 12b71069
>2012-12-17 10:20:21,878 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [13905503] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:20:30,014 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:20:30,029 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5126
>2012-12-17 10:20:30,031 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5126, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1a7a8e7a
>2012-12-17 10:20:30,054 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling started: taskId = a3f6f0da-7c97-4ce3-93a3-867b1921fdfd
>2012-12-17 10:20:31,076 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling ended: taskId = a3f6f0da-7c97-4ce3-93a3-867b1921fdfd task status = finished
>2012-12-17 10:20:31,094 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling ended, spm status: SPM
>2012-12-17 10:20:31,095 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a3f6f0da-7c97-4ce3-93a3-867b1921fdfd), log id: 368c5bb2
>2012-12-17 10:20:31,113 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, HSMClearTaskVDSCommand, log id: 368c5bb2
>2012-12-17 10:20:31,113 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@38d81f18, log id: 1a7a8e7a
>2012-12-17 10:20:31,805 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:20:31,997 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-13) [3d5285e1] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:20:31,997 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 28326016
>2012-12-17 10:20:32,018 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:20:32,269 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-4) [3f24993] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:20:32,322 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [3f24993] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3e71a3f8
>2012-12-17 10:20:32,323 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [3f24993] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 13e57dad
>2012-12-17 10:20:32,343 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [3f24993] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:32,493 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [3f24993] FINISH, SpmStopVDSCommand, log id: 13e57dad
>2012-12-17 10:20:32,507 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [3f24993] FINISH, ResetIrsVDSCommand, log id: 3e71a3f8
>2012-12-17 10:20:32,518 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [3f24993] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 63e3e292
>2012-12-17 10:20:32,518 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [3f24993] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:32,518 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [3f24993] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 63e3e292
>2012-12-17 10:20:32,578 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3d5285e1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:20:32,591 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3d5285e1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5127
>2012-12-17 10:20:32,593 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5127, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4586519d
>2012-12-17 10:20:32,615 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] spmStart polling started: taskId = 855dd530-fce8-46d8-ac4b-0a9c95002d56
>2012-12-17 10:20:33,638 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] spmStart polling ended: taskId = 855dd530-fce8-46d8-ac4b-0a9c95002d56 task status = finished
>2012-12-17 10:20:33,656 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] spmStart polling ended, spm status: SPM
>2012-12-17 10:20:33,658 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=855dd530-fce8-46d8-ac4b-0a9c95002d56), log id: 423987cf
>2012-12-17 10:20:33,676 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, HSMClearTaskVDSCommand, log id: 423987cf
>2012-12-17 10:20:33,676 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f034900, log id: 4586519d
>2012-12-17 10:20:33,689 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3d5285e1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:20:33,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:33,708 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 24432d9e
>2012-12-17 10:20:33,726 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 24432d9e
>2012-12-17 10:20:33,726 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 28326016
>2012-12-17 10:20:33,727 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-13) [3d5285e1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:20:33,727 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3667f2ab
>2012-12-17 10:20:33,727 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:33,728 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 75c6da8a
>2012-12-17 10:20:33,747 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 75c6da8a
>2012-12-17 10:20:33,747 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3667f2ab
>2012-12-17 10:20:33,747 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-13) [3d5285e1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:20:42,373 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68ede0fd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:20:42,656 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [2353b939] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:20:42,834 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [77748919] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:20:42,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [77748919] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2e770c49
>2012-12-17 10:20:42,895 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [77748919] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7d5a4ab7
>2012-12-17 10:20:42,915 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [77748919] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:43,063 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [77748919] FINISH, SpmStopVDSCommand, log id: 7d5a4ab7
>2012-12-17 10:20:43,080 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [77748919] FINISH, ResetIrsVDSCommand, log id: 2e770c49
>2012-12-17 10:20:43,091 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [77748919] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: e3a2344
>2012-12-17 10:20:43,091 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [77748919] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:43,091 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [77748919] FINISH, MarkPoolInReconstructModeVDSCommand, log id: e3a2344
>2012-12-17 10:20:52,658 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [742a6686] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:20:53,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [17997e65] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:20:53,159 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [17997e65] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5128
>2012-12-17 10:20:53,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [17997e65] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5128, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: b66d038
>2012-12-17 10:20:53,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [17997e65] spmStart polling started: taskId = eddc436d-94cd-4bdf-ada2-1fd4040a95ce
>2012-12-17 10:20:54,210 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [17997e65] spmStart polling ended: taskId = eddc436d-94cd-4bdf-ada2-1fd4040a95ce task status = finished
>2012-12-17 10:20:54,232 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [17997e65] spmStart polling ended, spm status: SPM
>2012-12-17 10:20:54,234 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [17997e65] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=eddc436d-94cd-4bdf-ada2-1fd4040a95ce), log id: 54af3d68
>2012-12-17 10:20:54,252 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [17997e65] FINISH, HSMClearTaskVDSCommand, log id: 54af3d68
>2012-12-17 10:20:54,252 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [17997e65] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3ce8a247, log id: b66d038
>2012-12-17 10:20:54,260 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [17997e65] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:20:54,320 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:20:54,320 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3a768402
>2012-12-17 10:20:54,349 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [17997e65] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:20:54,503 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [4201b256] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:20:54,648 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [4201b256] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2aa8f083
>2012-12-17 10:20:54,650 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [4201b256] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3134b1ca
>2012-12-17 10:20:54,670 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [4201b256] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:54,843 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [4201b256] FINISH, SpmStopVDSCommand, log id: 3134b1ca
>2012-12-17 10:20:54,870 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [4201b256] FINISH, ResetIrsVDSCommand, log id: 2aa8f083
>2012-12-17 10:20:54,882 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [4201b256] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c16cce5
>2012-12-17 10:20:54,882 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [4201b256] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:54,882 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [4201b256] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c16cce5
>2012-12-17 10:20:54,972 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:20:55,007 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5129
>2012-12-17 10:20:55,008 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5129, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 123ed952
>2012-12-17 10:20:55,033 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling started: taskId = bb7de44f-682e-4db6-961d-1315b85da3eb
>2012-12-17 10:20:56,058 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended: taskId = bb7de44f-682e-4db6-961d-1315b85da3eb task status = finished
>2012-12-17 10:20:56,079 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended, spm status: SPM
>2012-12-17 10:20:56,080 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bb7de44f-682e-4db6-961d-1315b85da3eb), log id: 8ff109a
>2012-12-17 10:20:56,099 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMClearTaskVDSCommand, log id: 8ff109a
>2012-12-17 10:20:56,099 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@219af313, log id: 123ed952
>2012-12-17 10:20:56,489 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:20:56,547 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:56,549 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 42ec0952
>2012-12-17 10:20:56,568 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 42ec0952
>2012-12-17 10:20:56,568 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3a768402
>2012-12-17 10:20:56,568 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:20:56,568 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 165808d1
>2012-12-17 10:20:56,568 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:20:56,570 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2135187
>2012-12-17 10:20:56,588 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2135187
>2012-12-17 10:20:56,588 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 165808d1
>2012-12-17 10:20:56,588 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:21:02,933 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:21:05,058 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:21:05,190 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-3) [661fd7d8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:21:05,292 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 20982c93
>2012-12-17 10:21:05,294 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 336d0970
>2012-12-17 10:21:05,313 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:21:05,437 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] FINISH, SpmStopVDSCommand, log id: 336d0970
>2012-12-17 10:21:05,468 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] FINISH, ResetIrsVDSCommand, log id: 20982c93
>2012-12-17 10:21:05,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4bba8904
>2012-12-17 10:21:05,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [661fd7d8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:21:05,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4bba8904
>2012-12-17 10:21:13,231 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:21:15,533 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [e1a392] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:21:15,555 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [e1a392] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5130
>2012-12-17 10:21:15,557 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e1a392] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5130, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1762c3eb
>2012-12-17 10:21:15,579 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e1a392] spmStart polling started: taskId = ac7841a2-3c41-439c-ad83-92f4266ca09c
>2012-12-17 10:21:16,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e1a392] spmStart polling ended: taskId = ac7841a2-3c41-439c-ad83-92f4266ca09c task status = finished
>2012-12-17 10:21:16,619 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e1a392] spmStart polling ended, spm status: SPM
>2012-12-17 10:21:16,620 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [e1a392] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ac7841a2-3c41-439c-ad83-92f4266ca09c), log id: 55c31f58
>2012-12-17 10:21:16,639 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [e1a392] FINISH, HSMClearTaskVDSCommand, log id: 55c31f58
>2012-12-17 10:21:16,639 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e1a392] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7d3ef664, log id: 1762c3eb
>2012-12-17 10:21:17,000 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [e1a392] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:21:17,197 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [3f5f0fad] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:21:17,197 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 441096c4
>2012-12-17 10:21:17,237 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [e1a392] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:21:17,429 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [77bd05d1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:21:17,498 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [77bd05d1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 460f3232
>2012-12-17 10:21:17,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [77bd05d1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6120de80
>2012-12-17 10:21:17,519 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [77bd05d1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:21:17,676 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [77bd05d1] FINISH, SpmStopVDSCommand, log id: 6120de80
>2012-12-17 10:21:17,727 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [77bd05d1] FINISH, ResetIrsVDSCommand, log id: 460f3232
>2012-12-17 10:21:17,740 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [77bd05d1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d28ec20
>2012-12-17 10:21:17,740 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [77bd05d1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:21:17,740 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [77bd05d1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d28ec20
>2012-12-17 10:21:17,905 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:21:17,932 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5131
>2012-12-17 10:21:17,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5131, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 799ebab7
>2012-12-17 10:21:17,956 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] spmStart polling started: taskId = d90d8bc7-7767-4b7e-8b72-6b5a60e87d9d
>2012-12-17 10:21:18,978 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] spmStart polling ended: taskId = d90d8bc7-7767-4b7e-8b72-6b5a60e87d9d task status = finished
>2012-12-17 10:21:18,997 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] spmStart polling ended, spm status: SPM
>2012-12-17 10:21:18,998 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d90d8bc7-7767-4b7e-8b72-6b5a60e87d9d), log id: 6dbf493a
>2012-12-17 10:21:19,016 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, HSMClearTaskVDSCommand, log id: 6dbf493a
>2012-12-17 10:21:19,016 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2960b720, log id: 799ebab7
>2012-12-17 10:21:19,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:21:19,061 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:21:19,065 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6f7250a
>2012-12-17 10:21:19,084 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f7250a
>2012-12-17 10:21:19,084 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 441096c4
>2012-12-17 10:21:19,084 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [3f5f0fad] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:21:19,084 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2454af72
>2012-12-17 10:21:19,084 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:21:19,086 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1c309d82
>2012-12-17 10:21:19,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1c309d82
>2012-12-17 10:21:19,104 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2454af72
>2012-12-17 10:21:19,104 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [3f5f0fad] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:21:23,469 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [42fa3e4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:21:27,968 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [4020b36c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:21:28,142 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [7d28dac8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:21:28,185 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [7d28dac8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 433d12e6
>2012-12-17 10:21:28,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [7d28dac8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 31eb959c
>2012-12-17 10:21:28,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [7d28dac8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:21:28,404 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [7d28dac8] FINISH, SpmStopVDSCommand, log id: 31eb959c
>2012-12-17 10:21:28,418 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [7d28dac8] FINISH, ResetIrsVDSCommand, log id: 433d12e6
>2012-12-17 10:21:28,429 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [7d28dac8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6688c6d
>2012-12-17 10:21:28,429 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [7d28dac8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:21:28,429 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [7d28dac8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6688c6d
>2012-12-17 10:21:33,725 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [2b3eea84] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:21:38,504 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:21:38,535 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5132
>2012-12-17 10:21:38,536 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5132, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e7b9d17
>2012-12-17 10:21:38,559 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling started: taskId = b827fcd2-a8f9-421b-ae06-53ef7ff56692
>2012-12-17 10:21:39,582 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling ended: taskId = b827fcd2-a8f9-421b-ae06-53ef7ff56692 task status = finished
>2012-12-17 10:21:39,600 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling ended, spm status: SPM
>2012-12-17 10:21:39,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b827fcd2-a8f9-421b-ae06-53ef7ff56692), log id: 2078e1c7
>2012-12-17 10:21:39,620 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, HSMClearTaskVDSCommand, log id: 2078e1c7
>2012-12-17 10:21:39,620 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@602ee078, log id: 3e7b9d17
>2012-12-17 10:21:39,636 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:21:39,687 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [1dd89be6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:21:39,687 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 41a69a06
>2012-12-17 10:21:39,713 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:21:39,908 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [499dcbe0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:21:40,023 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [499dcbe0] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2eb0f4e0
>2012-12-17 10:21:40,025 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [499dcbe0] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 723f44aa
>2012-12-17 10:21:40,045 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [499dcbe0] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:21:40,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [499dcbe0] FINISH, SpmStopVDSCommand, log id: 723f44aa
>2012-12-17 10:21:40,210 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [499dcbe0] FINISH, ResetIrsVDSCommand, log id: 2eb0f4e0
>2012-12-17 10:21:40,221 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [499dcbe0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3992fffd
>2012-12-17 10:21:40,221 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [499dcbe0] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:21:40,221 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [499dcbe0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3992fffd
>2012-12-17 10:21:40,302 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:21:40,407 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5133
>2012-12-17 10:21:40,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5133, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3fc42cc5
>2012-12-17 10:21:40,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] spmStart polling started: taskId = 54efb0ea-7fb2-4934-ae23-53bbf297c464
>2012-12-17 10:21:41,452 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] spmStart polling ended: taskId = 54efb0ea-7fb2-4934-ae23-53bbf297c464 task status = finished
>2012-12-17 10:21:41,470 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] spmStart polling ended, spm status: SPM
>2012-12-17 10:21:41,472 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=54efb0ea-7fb2-4934-ae23-53bbf297c464), log id: 4cf57c2f
>2012-12-17 10:21:41,491 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, HSMClearTaskVDSCommand, log id: 4cf57c2f
>2012-12-17 10:21:41,491 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@8ac1922, log id: 3fc42cc5
>2012-12-17 10:21:41,508 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:21:41,580 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:21:41,582 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 79b34030
>2012-12-17 10:21:41,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 79b34030
>2012-12-17 10:21:41,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 41a69a06
>2012-12-17 10:21:41,601 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [1dd89be6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:21:41,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 673b4dbb
>2012-12-17 10:21:41,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:21:41,602 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3827853
>2012-12-17 10:21:41,621 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3827853
>2012-12-17 10:21:41,621 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 673b4dbb
>2012-12-17 10:21:41,621 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [1dd89be6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:21:44,007 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [6c52cc2e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:21:50,344 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [3c8f0aba] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:21:50,457 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-88) [549524c7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:21:50,532 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-88) [549524c7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 57b9a028
>2012-12-17 10:21:50,534 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [549524c7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f8c7ced
>2012-12-17 10:21:50,554 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [549524c7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:21:50,688 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [549524c7] FINISH, SpmStopVDSCommand, log id: 4f8c7ced
>2012-12-17 10:21:50,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-88) [549524c7] FINISH, ResetIrsVDSCommand, log id: 57b9a028
>2012-12-17 10:21:50,709 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [549524c7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: cc980ae
>2012-12-17 10:21:50,709 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [549524c7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:21:50,709 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [549524c7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: cc980ae
>2012-12-17 10:21:54,226 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [661fd7d8] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:22:00,810 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:22:00,905 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5134
>2012-12-17 10:22:00,906 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5134, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2ed2127b
>2012-12-17 10:22:00,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] spmStart polling started: taskId = ab1ace5a-3589-4fb7-a050-84bba529d2c9
>2012-12-17 10:22:01,952 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] spmStart polling ended: taskId = ab1ace5a-3589-4fb7-a050-84bba529d2c9 task status = finished
>2012-12-17 10:22:01,971 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] spmStart polling ended, spm status: SPM
>2012-12-17 10:22:01,972 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ab1ace5a-3589-4fb7-a050-84bba529d2c9), log id: 1bb7f8ee
>2012-12-17 10:22:01,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] FINISH, HSMClearTaskVDSCommand, log id: 1bb7f8ee
>2012-12-17 10:22:01,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4b37df79, log id: 2ed2127b
>2012-12-17 10:22:02,008 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:22:02,070 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [b0e3bca] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:22:02,070 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1cae50b2
>2012-12-17 10:22:02,081 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:22:02,209 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [692ca847] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:22:02,271 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [692ca847] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 63353176
>2012-12-17 10:22:02,273 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [692ca847] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 742a1311
>2012-12-17 10:22:02,293 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [692ca847] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:02,451 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [692ca847] FINISH, SpmStopVDSCommand, log id: 742a1311
>2012-12-17 10:22:02,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [692ca847] FINISH, ResetIrsVDSCommand, log id: 63353176
>2012-12-17 10:22:02,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [692ca847] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24131433
>2012-12-17 10:22:02,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [692ca847] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:02,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [692ca847] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24131433
>2012-12-17 10:22:02,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [b0e3bca] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:22:02,589 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [b0e3bca] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5135
>2012-12-17 10:22:02,590 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5135, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 73aa7313
>2012-12-17 10:22:02,612 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] spmStart polling started: taskId = dc148bb0-82c6-4180-bc19-36db42c4e1f9
>2012-12-17 10:22:03,634 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] spmStart polling ended: taskId = dc148bb0-82c6-4180-bc19-36db42c4e1f9 task status = finished
>2012-12-17 10:22:03,652 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] spmStart polling ended, spm status: SPM
>2012-12-17 10:22:03,653 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dc148bb0-82c6-4180-bc19-36db42c4e1f9), log id: 75acaad3
>2012-12-17 10:22:03,673 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, HSMClearTaskVDSCommand, log id: 75acaad3
>2012-12-17 10:22:03,673 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1b033f00, log id: 73aa7313
>2012-12-17 10:22:03,719 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [b0e3bca] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:22:03,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:03,744 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: daaa0d0
>2012-12-17 10:22:03,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: daaa0d0
>2012-12-17 10:22:03,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1cae50b2
>2012-12-17 10:22:03,765 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [b0e3bca] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:22:03,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3d23beb2
>2012-12-17 10:22:03,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:03,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 31dfa6f4
>2012-12-17 10:22:03,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31dfa6f4
>2012-12-17 10:22:03,785 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3d23beb2
>2012-12-17 10:22:03,785 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [b0e3bca] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:22:04,440 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:22:12,633 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [729d4e04] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:22:12,756 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [5c7113d1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:22:12,821 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [5c7113d1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 30f9728d
>2012-12-17 10:22:12,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [5c7113d1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5474c09d
>2012-12-17 10:22:12,842 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [5c7113d1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:13,012 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [5c7113d1] FINISH, SpmStopVDSCommand, log id: 5474c09d
>2012-12-17 10:22:13,041 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [5c7113d1] FINISH, ResetIrsVDSCommand, log id: 30f9728d
>2012-12-17 10:22:13,053 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [5c7113d1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7fba848d
>2012-12-17 10:22:13,053 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [5c7113d1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:13,053 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [5c7113d1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7fba848d
>2012-12-17 10:22:14,680 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [406c9a80] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:22:23,135 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [76c4364c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:22:23,147 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [76c4364c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5136
>2012-12-17 10:22:23,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5136, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7539f35f
>2012-12-17 10:22:23,171 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] spmStart polling started: taskId = 741a28ce-edb3-4672-b36d-af31fcbcb7b7
>2012-12-17 10:22:24,198 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] spmStart polling ended: taskId = 741a28ce-edb3-4672-b36d-af31fcbcb7b7 task status = finished
>2012-12-17 10:22:24,216 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] spmStart polling ended, spm status: SPM
>2012-12-17 10:22:24,217 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=741a28ce-edb3-4672-b36d-af31fcbcb7b7), log id: 62c5f928
>2012-12-17 10:22:24,236 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] FINISH, HSMClearTaskVDSCommand, log id: 62c5f928
>2012-12-17 10:22:24,236 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3368e3c1, log id: 7539f35f
>2012-12-17 10:22:24,246 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [76c4364c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:22:24,309 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:22:24,309 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 43ee7ab1
>2012-12-17 10:22:24,324 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [76c4364c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:22:24,427 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [56be0aeb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:22:24,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [56be0aeb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 25028416
>2012-12-17 10:22:24,482 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [56be0aeb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 17107575
>2012-12-17 10:22:24,502 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [56be0aeb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:24,611 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [56be0aeb] FINISH, SpmStopVDSCommand, log id: 17107575
>2012-12-17 10:22:24,625 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [56be0aeb] FINISH, ResetIrsVDSCommand, log id: 25028416
>2012-12-17 10:22:24,636 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [56be0aeb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7eaefa6c
>2012-12-17 10:22:24,636 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [56be0aeb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:24,636 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [56be0aeb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7eaefa6c
>2012-12-17 10:22:24,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:22:24,756 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5137
>2012-12-17 10:22:24,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5137, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7809d515
>2012-12-17 10:22:24,783 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling started: taskId = f3e5a714-ea4c-4902-b133-38f0a98c247c
>2012-12-17 10:22:25,805 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling ended: taskId = f3e5a714-ea4c-4902-b133-38f0a98c247c task status = finished
>2012-12-17 10:22:25,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling ended, spm status: SPM
>2012-12-17 10:22:25,824 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f3e5a714-ea4c-4902-b133-38f0a98c247c), log id: 79e05bea
>2012-12-17 10:22:25,843 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMClearTaskVDSCommand, log id: 79e05bea
>2012-12-17 10:22:25,843 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78b29e28, log id: 7809d515
>2012-12-17 10:22:26,379 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:22:26,642 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:26,646 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 323099d8
>2012-12-17 10:22:26,667 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 323099d8
>2012-12-17 10:22:26,667 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 43ee7ab1
>2012-12-17 10:22:26,668 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:22:26,668 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 14c44da6
>2012-12-17 10:22:26,668 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:26,669 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 51ebb28b
>2012-12-17 10:22:26,688 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 51ebb28b
>2012-12-17 10:22:26,688 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 14c44da6
>2012-12-17 10:22:26,688 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:22:34,782 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [79d25c3b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:22:34,882 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [d5f200c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:22:34,939 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [d5f200c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6a750d19
>2012-12-17 10:22:34,941 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [d5f200c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7a7ed3d7
>2012-12-17 10:22:34,962 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [d5f200c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:35,119 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [565ac2be] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:22:35,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [d5f200c] FINISH, SpmStopVDSCommand, log id: 7a7ed3d7
>2012-12-17 10:22:35,161 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [d5f200c] FINISH, ResetIrsVDSCommand, log id: 6a750d19
>2012-12-17 10:22:35,192 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [d5f200c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1fc2e3fb
>2012-12-17 10:22:35,192 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [d5f200c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:35,192 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [d5f200c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1fc2e3fb
>2012-12-17 10:22:45,255 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:22:45,267 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5138
>2012-12-17 10:22:45,268 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5138, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 11ede958
>2012-12-17 10:22:45,292 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling started: taskId = a1dde10b-3bb1-4d8c-ab63-d77d2c4ecf22
>2012-12-17 10:22:46,314 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling ended: taskId = a1dde10b-3bb1-4d8c-ab63-d77d2c4ecf22 task status = finished
>2012-12-17 10:22:46,334 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling ended, spm status: SPM
>2012-12-17 10:22:46,335 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a1dde10b-3bb1-4d8c-ab63-d77d2c4ecf22), log id: 32eb168e
>2012-12-17 10:22:46,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMClearTaskVDSCommand, log id: 32eb168e
>2012-12-17 10:22:46,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6815debf, log id: 11ede958
>2012-12-17 10:22:46,452 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:22:46,514 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [60aea922] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:22:46,514 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 9ed208b
>2012-12-17 10:22:46,531 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:22:46,687 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [5f08853c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:22:46,762 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [5f08853c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2319934b
>2012-12-17 10:22:46,764 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [5f08853c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 62da6a01
>2012-12-17 10:22:46,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [5f08853c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:46,927 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [5f08853c] FINISH, SpmStopVDSCommand, log id: 62da6a01
>2012-12-17 10:22:46,957 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [5f08853c] FINISH, ResetIrsVDSCommand, log id: 2319934b
>2012-12-17 10:22:46,968 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [5f08853c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 47ee91e8
>2012-12-17 10:22:46,968 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5f08853c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:46,968 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [5f08853c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 47ee91e8
>2012-12-17 10:22:47,022 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [60aea922] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:22:47,032 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [60aea922] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5139
>2012-12-17 10:22:47,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5139, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7295286
>2012-12-17 10:22:47,065 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [60aea922] spmStart polling started: taskId = 5a65cda4-2a13-4c41-a23e-610495f75743
>2012-12-17 10:22:48,087 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [60aea922] spmStart polling ended: taskId = 5a65cda4-2a13-4c41-a23e-610495f75743 task status = finished
>2012-12-17 10:22:48,107 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [60aea922] spmStart polling ended, spm status: SPM
>2012-12-17 10:22:48,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5a65cda4-2a13-4c41-a23e-610495f75743), log id: 42ea22ef
>2012-12-17 10:22:48,127 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, HSMClearTaskVDSCommand, log id: 42ea22ef
>2012-12-17 10:22:48,127 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@199b9b3b, log id: 7295286
>2012-12-17 10:22:48,153 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [60aea922] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:22:48,173 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:48,175 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 22a9285f
>2012-12-17 10:22:48,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 22a9285f
>2012-12-17 10:22:48,207 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 9ed208b
>2012-12-17 10:22:48,207 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [60aea922] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:22:48,208 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 285daf40
>2012-12-17 10:22:48,208 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:48,209 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 74c411b8
>2012-12-17 10:22:48,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 74c411b8
>2012-12-17 10:22:48,239 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 285daf40
>2012-12-17 10:22:48,239 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [60aea922] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:22:55,564 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [3f24993] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:22:57,078 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [8102db] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:22:57,328 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [59abfc06] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:22:57,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [59abfc06] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: daf5a69
>2012-12-17 10:22:57,416 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [59abfc06] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2171236b
>2012-12-17 10:22:57,435 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [59abfc06] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:57,594 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [59abfc06] FINISH, SpmStopVDSCommand, log id: 2171236b
>2012-12-17 10:22:57,628 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [59abfc06] FINISH, ResetIrsVDSCommand, log id: daf5a69
>2012-12-17 10:22:57,639 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [59abfc06] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 262d12fa
>2012-12-17 10:22:57,639 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [59abfc06] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:22:57,639 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [59abfc06] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 262d12fa
>2012-12-17 10:23:03,705 INFO  [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-3) Checking if user admin@internal is an admin, result true
>2012-12-17 10:23:03,706 INFO  [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginAdminUserCommand internal: false.
>2012-12-17 10:23:06,545 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68ede0fd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:23:07,750 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:23:07,814 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5140
>2012-12-17 10:23:07,815 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5140, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1b4ad868
>2012-12-17 10:23:07,838 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling started: taskId = 50cf4448-8014-4cf9-a98f-34e30f82b4e0
>2012-12-17 10:23:08,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended: taskId = 50cf4448-8014-4cf9-a98f-34e30f82b4e0 task status = finished
>2012-12-17 10:23:08,878 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended, spm status: SPM
>2012-12-17 10:23:08,879 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=50cf4448-8014-4cf9-a98f-34e30f82b4e0), log id: 37cdef84
>2012-12-17 10:23:08,897 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMClearTaskVDSCommand, log id: 37cdef84
>2012-12-17 10:23:08,897 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3127dfbb, log id: 1b4ad868
>2012-12-17 10:23:08,935 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:23:08,998 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-29) [2b3eea84] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:23:08,998 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 28418e7e
>2012-12-17 10:23:08,998 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:23:09,112 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-12) [49d26cb8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:23:09,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [49d26cb8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1e3042fa
>2012-12-17 10:23:09,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [49d26cb8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1f3563d3
>2012-12-17 10:23:09,180 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [49d26cb8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:09,356 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [49d26cb8] FINISH, SpmStopVDSCommand, log id: 1f3563d3
>2012-12-17 10:23:09,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [49d26cb8] FINISH, ResetIrsVDSCommand, log id: 1e3042fa
>2012-12-17 10:23:09,379 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [49d26cb8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7220b4cc
>2012-12-17 10:23:09,379 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [49d26cb8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:09,379 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [49d26cb8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7220b4cc
>2012-12-17 10:23:09,447 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [2b3eea84] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:23:09,463 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [2b3eea84] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5141
>2012-12-17 10:23:09,464 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5141, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1f413fca
>2012-12-17 10:23:09,486 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] spmStart polling started: taskId = 56283b57-fbe4-4f33-8767-0be7deea3156
>2012-12-17 10:23:10,508 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] spmStart polling ended: taskId = 56283b57-fbe4-4f33-8767-0be7deea3156 task status = finished
>2012-12-17 10:23:10,526 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] spmStart polling ended, spm status: SPM
>2012-12-17 10:23:10,527 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=56283b57-fbe4-4f33-8767-0be7deea3156), log id: 4e2b9f95
>2012-12-17 10:23:10,547 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, HSMClearTaskVDSCommand, log id: 4e2b9f95
>2012-12-17 10:23:10,547 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@11fa174b, log id: 1f413fca
>2012-12-17 10:23:10,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [2b3eea84] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:23:10,594 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:10,598 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 75fa936f
>2012-12-17 10:23:10,618 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 75fa936f
>2012-12-17 10:23:10,619 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 28418e7e
>2012-12-17 10:23:10,619 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-29) [2b3eea84] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:23:10,619 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 15e598ec
>2012-12-17 10:23:10,619 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:10,620 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 9ebd604
>2012-12-17 10:23:10,640 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 9ebd604
>2012-12-17 10:23:10,640 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 15e598ec
>2012-12-17 10:23:10,640 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-29) [2b3eea84] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:23:16,802 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:23:19,495 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [454eb5a9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:23:19,669 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-68) [4ba100c3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:23:19,754 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [4ba100c3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1a02fcd1
>2012-12-17 10:23:19,756 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [4ba100c3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2c37892f
>2012-12-17 10:23:19,778 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [4ba100c3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:19,937 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [4ba100c3] FINISH, SpmStopVDSCommand, log id: 2c37892f
>2012-12-17 10:23:20,031 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [4ba100c3] FINISH, ResetIrsVDSCommand, log id: 1a02fcd1
>2012-12-17 10:23:20,042 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [4ba100c3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24634c5e
>2012-12-17 10:23:20,042 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [4ba100c3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:20,042 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [4ba100c3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24634c5e
>2012-12-17 10:23:27,077 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [102f58ae] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:23:30,126 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [661fd7d8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:23:30,142 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [661fd7d8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5142
>2012-12-17 10:23:30,144 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5142, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 18f9e88d
>2012-12-17 10:23:30,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] spmStart polling started: taskId = 1dbd1b3d-bb9c-4539-8d4f-ae1d4196f8ea
>2012-12-17 10:23:31,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] spmStart polling ended: taskId = 1dbd1b3d-bb9c-4539-8d4f-ae1d4196f8ea task status = finished
>2012-12-17 10:23:31,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] spmStart polling ended, spm status: SPM
>2012-12-17 10:23:31,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1dbd1b3d-bb9c-4539-8d4f-ae1d4196f8ea), log id: 4bcb844e
>2012-12-17 10:23:31,226 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] FINISH, HSMClearTaskVDSCommand, log id: 4bcb844e
>2012-12-17 10:23:31,226 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1e5b43c2, log id: 18f9e88d
>2012-12-17 10:23:31,416 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [661fd7d8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:23:31,720 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [65d7a0e6] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:23:31,720 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 27cb0408
>2012-12-17 10:23:31,783 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [661fd7d8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:23:31,966 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-3) [3118271c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:23:32,155 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [3118271c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 59d594b3
>2012-12-17 10:23:32,156 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [3118271c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 335a21ee
>2012-12-17 10:23:32,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [3118271c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:32,330 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [3118271c] FINISH, SpmStopVDSCommand, log id: 335a21ee
>2012-12-17 10:23:32,350 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [3118271c] FINISH, ResetIrsVDSCommand, log id: 59d594b3
>2012-12-17 10:23:32,361 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [3118271c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 140aed25
>2012-12-17 10:23:32,361 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [3118271c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:32,361 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [3118271c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 140aed25
>2012-12-17 10:23:32,431 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [65d7a0e6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:23:32,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [65d7a0e6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5143
>2012-12-17 10:23:32,474 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5143, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 798f9718
>2012-12-17 10:23:32,498 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] spmStart polling started: taskId = 70904064-a848-40ba-bb75-e754a51088fa
>2012-12-17 10:23:33,520 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] spmStart polling ended: taskId = 70904064-a848-40ba-bb75-e754a51088fa task status = finished
>2012-12-17 10:23:33,539 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] spmStart polling ended, spm status: SPM
>2012-12-17 10:23:33,540 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=70904064-a848-40ba-bb75-e754a51088fa), log id: 558ac7f6
>2012-12-17 10:23:33,559 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, HSMClearTaskVDSCommand, log id: 558ac7f6
>2012-12-17 10:23:33,559 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@a4e135a, log id: 798f9718
>2012-12-17 10:23:33,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [65d7a0e6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:23:33,605 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:33,607 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7eba84ab
>2012-12-17 10:23:33,626 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7eba84ab
>2012-12-17 10:23:33,626 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 27cb0408
>2012-12-17 10:23:33,626 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [65d7a0e6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:23:33,627 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 75c378d
>2012-12-17 10:23:33,627 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:33,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2440936
>2012-12-17 10:23:33,648 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2440936
>2012-12-17 10:23:33,648 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 75c378d
>2012-12-17 10:23:33,648 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [65d7a0e6] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:23:37,280 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [30ddd635] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:23:42,488 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [1290f5b5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:23:42,776 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-99) [24448a03] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:23:42,840 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [24448a03] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6e36de98
>2012-12-17 10:23:42,841 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [24448a03] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 441d3e65
>2012-12-17 10:23:42,862 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [24448a03] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:42,989 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [24448a03] FINISH, SpmStopVDSCommand, log id: 441d3e65
>2012-12-17 10:23:43,029 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [24448a03] FINISH, ResetIrsVDSCommand, log id: 6e36de98
>2012-12-17 10:23:43,040 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [24448a03] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6bdfa4c6
>2012-12-17 10:23:43,040 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [24448a03] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:43,040 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [24448a03] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6bdfa4c6
>2012-12-17 10:23:47,693 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [681720a4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:23:53,138 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7aeab1da] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:23:53,166 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7aeab1da] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5144
>2012-12-17 10:23:53,168 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5144, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1c10d5b8
>2012-12-17 10:23:53,190 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] spmStart polling started: taskId = 66bda7d6-1474-437f-bb59-02b7b2b420cf
>2012-12-17 10:23:54,211 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] spmStart polling ended: taskId = 66bda7d6-1474-437f-bb59-02b7b2b420cf task status = finished
>2012-12-17 10:23:54,236 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] spmStart polling ended, spm status: SPM
>2012-12-17 10:23:54,237 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=66bda7d6-1474-437f-bb59-02b7b2b420cf), log id: 4ccc650d
>2012-12-17 10:23:54,256 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] FINISH, HSMClearTaskVDSCommand, log id: 4ccc650d
>2012-12-17 10:23:54,256 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7efe5fde, log id: 1c10d5b8
>2012-12-17 10:23:54,345 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7aeab1da] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:23:54,423 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Attempting to get and stop tasks on storage pool TEST-DC
>2012-12-17 10:23:54,424 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4a4d6a62
>2012-12-17 10:23:54,439 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7aeab1da] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:23:54,672 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [2144ca30] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:23:54,723 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [2144ca30] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 667b98ad
>2012-12-17 10:23:54,725 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [2144ca30] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 67111f83
>2012-12-17 10:23:54,749 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [2144ca30] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:54,898 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [2144ca30] FINISH, SpmStopVDSCommand, log id: 67111f83
>2012-12-17 10:23:54,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [2144ca30] FINISH, ResetIrsVDSCommand, log id: 667b98ad
>2012-12-17 10:23:54,932 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [2144ca30] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3083d35c
>2012-12-17 10:23:54,932 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [2144ca30] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:54,932 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [2144ca30] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3083d35c
>2012-12-17 10:23:54,980 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC
>2012-12-17 10:23:55,005 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5145
>2012-12-17 10:23:55,006 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5145, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7e329617
>2012-12-17 10:23:55,029 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling started: taskId = 2b0eee60-ab04-49a7-aeec-7ac50dc3b87c
>2012-12-17 10:23:56,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling ended: taskId = 2b0eee60-ab04-49a7-aeec-7ac50dc3b87c task status = finished
>2012-12-17 10:23:56,070 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling ended, spm status: SPM
>2012-12-17 10:23:56,072 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2b0eee60-ab04-49a7-aeec-7ac50dc3b87c), log id: 33183034
>2012-12-17 10:23:56,091 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMClearTaskVDSCommand, log id: 33183034
>2012-12-17 10:23:56,091 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7904c020, log id: 7e329617
>2012-12-17 10:23:56,780 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com
>2012-12-17 10:23:56,862 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:56,863 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7ce42015
>2012-12-17 10:23:56,886 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7ce42015
>2012-12-17 10:23:56,886 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4a4d6a62
>2012-12-17 10:23:56,886 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:23:56,886 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 504e9ca9
>2012-12-17 10:23:56,886 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:23:56,887 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 685dcd3f
>2012-12-17 10:23:56,906 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 685dcd3f
>2012-12-17 10:23:56,907 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 504e9ca9
>2012-12-17 10:23:56,907 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Discovered no tasks on Storage Pool TEST-DC
>2012-12-17 10:23:58,337 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [36ca77a1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:24:05,017 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [b18376] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2
>2012-12-17 10:24:05,119 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [7f5c814d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage
>2012-12-17 10:24:05,209 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [7f5c814d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 69aeca7c
>2012-12-17 10:24:05,211 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [7f5c814d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f07adff
>2012-12-17 10:24:05,235 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [7f5c814d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:24:05,374 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [7f5c814d] FINISH, SpmStopVDSCommand, log id: 4f07adff
>2012-12-17 10:24:05,405 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [7f5c814d] FINISH, ResetIrsVDSCommand, log id: 69aeca7c
>2012-12-17 10:24:05,416 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [7f5c814d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3e52a2e2
>2012-12-17 10:24:05,416 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [7f5c814d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd
>2012-12-17 10:24:05,416 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [7f5c814d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3e52a2e2
>2012-12-17 10:24:08,559 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status
>2012-12-17 10:24:11,871 INFO  [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-3-thread-50) [46708fc2] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected :  ID: 8ed2188e-4795-11e2-b963-001122334455 Type: VDS
>2012-12-17 10:24:11,873 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-50) [46708fc2] START, SetVdsStatusVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 2f9d8b07
>2012-12-17 10:24:11,916 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-50) [46708fc2] FINISH, SetVdsStatusVDSCommand, log id: 2f9d8b07
>2012-12-17 10:24:12,002 INFO  [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-3-thread-50) [46708fc2] Running command: MaintananceVdsCommand internal: true. Entities affected :  ID: 8ed2188e-4795-11e2-b963-001122334455 Type: VDS
>2012-12-17 10:24:12,636 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-55) [4007c40f] vds::Updated vds status from Preparing for Maintenance to Maintenance in database,  vds = 8ed2188e-4795-11e2-b963-001122334455 : pink-vds2
>2012-12-17 10:24:12,794 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] Clearing cache of pool: 03bf4661-cd9b-4581-a18f-ed7f7be148dd for problematic entities of VDS: pink-vds2.
>2012-12-17 10:24:12,796 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, DisconnectStoragePoolVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, vds_spm_id = 1), log id: 73fd3692
>2012-12-17 10:24:14,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, DisconnectStoragePoolVDSCommand, log id: 73fd3692
>2012-12-17 10:24:14,825 INFO  [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (QuartzScheduler_Worker-55) [71356416] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected :  ID: 03bf4661-cd9b-4581-a18f-ed7f7be148dd Type: StoragePool
>2012-12-17 10:24:14,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (QuartzScheduler_Worker-55) [71356416] START, DisconnectStorageServerVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, storageType = NFS, connectionList = [{ id: e6b2daa0-82fe-4c87-af44-a7017d09cdca, connection: orion.qa.lab.tlv.redhat.com:/export/gena/st_3000, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 9d5d0cd
>2012-12-17 10:24:18,044 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (QuartzScheduler_Worker-55) [71356416] FINISH, DisconnectStorageServerVDSCommand, return: {e6b2daa0-82fe-4c87-af44-a7017d09cdca=0}, log id: 9d5d0cd
>2012-12-17 10:25:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [50119c91] Checking autorecoverable hosts
>2012-12-17 10:25:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [50119c91] Autorecovering 0 hosts
>2012-12-17 10:25:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [50119c91] Checking autorecoverable hosts done
>2012-12-17 10:25:00,003 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [50119c91] Autorecovering storage domains is disabled, skipping
>2012-12-17 10:25:51,779 INFO  [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-58) [5cdf4906] DbUserCacheManager::refreshAllUserData() - entered
>2012-12-17 10:30:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [53480d94] Checking autorecoverable hosts
>2012-12-17 10:30:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [53480d94] Autorecovering 0 hosts
>2012-12-17 10:30:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [53480d94] Checking autorecoverable hosts done
>2012-12-17 10:30:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [53480d94] Autorecovering storage domains is disabled, skipping
>2012-12-17 10:34:15,610 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-19) [4fc51f8f] vds::refreshVdsStats Failed getVdsStats,  vds = 39a7379e-439a-11e2-ba16-001122334455 : orchid-vds2, error = VDSNetworkException: VDSNetworkException: 
>2012-12-17 10:34:15,636 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-19) [4fc51f8f] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 39a7379e-439a-11e2-ba16-001122334455 : orchid-vds2, VDS Network Error, continuing.
>VDSNetworkException: 
>2012-12-17 10:34:17,644 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-37) [24cb4742] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 39a7379e-439a-11e2-ba16-001122334455 : orchid-vds2, VDS Network Error, continuing.
>VDSNetworkException: 
>2012-12-17 10:34:19,652 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-8) [536ede7] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 39a7379e-439a-11e2-ba16-001122334455 : orchid-vds2, VDS Network Error, continuing.
>VDSNetworkException: 
>2012-12-17 10:34:20,805 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-20) [76ee6c44] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 27ecc06e-1f3c-4470-ac7d-94740ed00e96 Type: StoragePool
>2012-12-17 10:34:20,911 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [76ee6c44] hostFromVds::selectedVds - orchid-vds1, spmStatus Free, storage pool CL30
>2012-12-17 10:34:20,936 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [76ee6c44] SPM Init: could not find reported vds or not up - pool:CL30 vds_spm_id: 2
>2012-12-17 10:34:20,938 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [76ee6c44] SPM selection - vds seems as spm orchid-vds2
>2012-12-17 10:34:20,939 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [76ee6c44] START, SpmStopVDSCommand(HostName = orchid-vds2, HostId = 39a7379e-439a-11e2-ba16-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96), log id: 76cc1ce6
>2012-12-17 10:34:20,939 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [76ee6c44] SpmStopVDSCommand:: vds orchid-vds2 is in Connecting status - not performing spm stop, pool id 27ecc06e-1f3c-4470-ac7d-94740ed00e96
>2012-12-17 10:34:20,939 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [76ee6c44] FINISH, SpmStopVDSCommand, log id: 76cc1ce6
>2012-12-17 10:34:20,939 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [76ee6c44] spm stop on spm failed, stopping spm selection!
>2012-12-17 10:34:22,703 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-36) [7b7a7712] Running command: InitVdsOnUpCommand internal: true.
>2012-12-17 10:34:22,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-36) [4464e415] START, ValidateStorageServerConnectionVDSCommand(HostName = orchid-vds2, HostId = 39a7379e-439a-11e2-ba16-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, storageType = NFS, connectionList = [{ id: 696c5b0d-330e-4667-89c7-428563d6fc47, connection: orion.qa.lab.tlv.redhat.com:/export/meni/upgrade, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 662c11c0
>2012-12-17 10:34:22,811 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-36) [4464e415] FINISH, ValidateStorageServerConnectionVDSCommand, return: {696c5b0d-330e-4667-89c7-428563d6fc47=0}, log id: 662c11c0
>2012-12-17 10:34:22,812 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-36) [4464e415] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 27ecc06e-1f3c-4470-ac7d-94740ed00e96 Type: StoragePool
>2012-12-17 10:34:22,813 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-36) [4464e415] START, ConnectStorageServerVDSCommand(HostName = orchid-vds2, HostId = 39a7379e-439a-11e2-ba16-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, storageType = NFS, connectionList = [{ id: 696c5b0d-330e-4667-89c7-428563d6fc47, connection: orion.qa.lab.tlv.redhat.com:/export/meni/upgrade, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4263722c
>2012-12-17 10:34:22,835 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-36) [4464e415] FINISH, ConnectStorageServerVDSCommand, return: {696c5b0d-330e-4667-89c7-428563d6fc47=0}, log id: 4263722c
>2012-12-17 10:34:22,835 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-36) [4464e415] Host orchid-vds2 storage connection was succeeded 
>2012-12-17 10:34:22,855 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-36) [4464e415] START, ConnectStoragePoolVDSCommand(HostName = orchid-vds2, HostId = 39a7379e-439a-11e2-ba16-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, vds_spm_id = 2, masterDomainId = e56c6a97-fb9a-4e87-b77b-046bd7f29be0, masterVersion = 1), log id: 273d2fd7
>2012-12-17 10:34:28,111 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-36) [4464e415] FINISH, ConnectStoragePoolVDSCommand, log id: 273d2fd7
>2012-12-17 10:34:28,174 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [4464e415] No string for UNASSIGNED type. Use default Log
>2012-12-17 10:34:28,177 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-47) [4464e415] MultipleActionsRunner of type MigrateVmToServer invoked with no actions
>2012-12-17 10:34:28,206 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [4ee87fc2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 39a7379e-439a-11e2-ba16-001122334455 Type: VDS
>2012-12-17 10:34:28,227 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [7b61ca13] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 39a7379e-439a-11e2-ba16-001122334455 Type: VDS
>2012-12-17 10:34:30,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2d5b99a4] hostFromVds::selectedVds - orchid-vds1, spmStatus Free, storage pool CL30
>2012-12-17 10:34:30,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2d5b99a4] Found spm host 39a7379e-439a-11e2-ba16-001122334455, host name: orchid-vds2, according to spmId: 2.
>2012-12-17 10:34:30,973 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2d5b99a4] SpmStatus on vds 39a7379e-439a-11e2-ba16-001122334455: Free
>2012-12-17 10:34:31,636 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2d5b99a4] starting spm on vds orchid-vds1, storage pool CL30, prevId 2, LVER 0
>2012-12-17 10:34:31,638 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] START, SpmStartVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, prevId=2, prevLVER=0, storagePoolFormatType=V1, recoveryMode=Manual, SCSIFencing=false), log id: 2db22385
>2012-12-17 10:34:31,650 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] spmStart polling started: taskId = 98e27a6d-d465-4e7e-8607-726b183f76ac
>2012-12-17 10:34:53,881 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] spmStart polling ended: taskId = 98e27a6d-d465-4e7e-8607-726b183f76ac task status = finished
>2012-12-17 10:34:53,886 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] spmStart polling ended, spm status: SPM
>2012-12-17 10:34:53,887 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] START, HSMClearTaskVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, taskId=98e27a6d-d465-4e7e-8607-726b183f76ac), log id: 49a8a30
>2012-12-17 10:34:53,892 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] FINISH, HSMClearTaskVDSCommand, log id: 49a8a30
>2012-12-17 10:34:53,892 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3de657ac, log id: 2db22385
>2012-12-17 10:34:53,921 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2d5b99a4] Initialize Irs proxy from vds: orchid-vds1.qa.lab.tlv.redhat.com
>2012-12-17 10:34:53,969 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [565ac2be] Attempting to get and stop tasks on storage pool CL30
>2012-12-17 10:34:53,969 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 324aa7e6
>2012-12-17 10:34:54,061 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 27ecc06e-1f3c-4470-ac7d-94740ed00e96
>2012-12-17 10:34:54,063 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] START, HSMGetAllTasksInfoVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455), log id: 5622bea0
>2012-12-17 10:34:54,068 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5622bea0
>2012-12-17 10:34:54,068 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 324aa7e6
>2012-12-17 10:34:54,068 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [565ac2be] Discovered no tasks on Storage Pool CL30
>2012-12-17 10:34:54,068 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 311268e
>2012-12-17 10:34:54,068 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 27ecc06e-1f3c-4470-ac7d-94740ed00e96
>2012-12-17 10:34:54,069 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] START, HSMGetAllTasksInfoVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455), log id: 703af56a
>2012-12-17 10:34:54,075 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 703af56a
>2012-12-17 10:34:54,075 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 311268e
>2012-12-17 10:34:54,075 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [565ac2be] Discovered no tasks on Storage Pool CL30
>2012-12-17 10:35:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [57fa1ec] Checking autorecoverable hosts
>2012-12-17 10:35:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [57fa1ec] Autorecovering 0 hosts
>2012-12-17 10:35:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [57fa1ec] Checking autorecoverable hosts done
>2012-12-17 10:35:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [57fa1ec] Autorecovering storage domains is disabled, skipping
>2012-12-17 10:36:00,109 INFO  [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-3-thread-48) [e712868] Running command: MigrateVmCommand internal: false. Entities affected :  ID: e9e35b3d-031d-4ef3-a8b4-4fd83936b75b Type: VM
>2012-12-17 10:36:00,120 INFO  [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-3-thread-48) [e712868] START, MigrateVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, srcHost=orchid-vds1.qa.lab.tlv.redhat.com, dstVdsId=39a7379e-439a-11e2-ba16-001122334455, dstHost=orchid-vds2.qa.lab.tlv.redhat.com:54321, migrationMethod=ONLINE), log id: 7bb4865f
>2012-12-17 10:36:00,123 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-3-thread-48) [e712868] VdsBroker::migrate::Entered (vm_guid=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, srcHost=orchid-vds1.qa.lab.tlv.redhat.com, dstHost=orchid-vds2.qa.lab.tlv.redhat.com:54321,  method=online
>2012-12-17 10:36:00,123 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-3-thread-48) [e712868] START, MigrateBrokerVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, srcHost=orchid-vds1.qa.lab.tlv.redhat.com, dstVdsId=39a7379e-439a-11e2-ba16-001122334455, dstHost=orchid-vds2.qa.lab.tlv.redhat.com:54321, migrationMethod=ONLINE), log id: 41afcf86
>2012-12-17 10:36:00,131 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-3-thread-48) [e712868] FINISH, MigrateBrokerVDSCommand, log id: 41afcf86
>2012-12-17 10:36:00,164 INFO  [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-3-thread-48) [e712868] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7bb4865f
>2012-12-17 10:36:00,574 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-20) [76ee6c44] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:02,627 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-13) [3d5285e1] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:04,678 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-27) [956a2f9] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:06,900 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [5c1b0caf] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:08,945 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-6) [5c7113d1] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:10,987 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-26) [406c9a80] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:13,032 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-64) [77748919] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:15,080 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-25) [127fd955] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:17,211 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-29) [2b3eea84] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:19,255 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-17) [56be0aeb] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:21,295 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-41) [25e2aebf] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:23,340 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-57) [499dcbe0] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:25,388 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) [7f5c814d] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:27,495 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-96) [50119c91] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:29,539 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-2) [d5f200c] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:31,581 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) [1630089b] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:36:33,434 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-88) [549524c7] START, DestroyVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, force=false, secondsToWait=0, gracefully=false), log id: 420b884a
>2012-12-17 10:36:33,511 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-88) [549524c7] FINISH, DestroyVDSCommand, log id: 420b884a
>2012-12-17 10:36:33,512 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-88) [549524c7] Setting VM vm_30 e9e35b3d-031d-4ef3-a8b4-4fd83936b75b to status unknown
>2012-12-17 10:36:33,512 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-88) [549524c7] vm vm_30 running in db and not running in vds - add to rerun treatment. vds orchid-vds1
>2012-12-17 10:37:52,032 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-64) [77748919] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 7d08a922-3d3d-11e2-82f1-001122334455 : orchid-vds1, VDS Network Error, continuing.
>VDSNetworkException: 
>2012-12-17 10:37:54,050 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-12) [49d26cb8] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 7d08a922-3d3d-11e2-82f1-001122334455 : orchid-vds1, VDS Network Error, continuing.
>VDSNetworkException: 
>2012-12-17 10:37:55,791 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-28) [3484e0f5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 27ecc06e-1f3c-4470-ac7d-94740ed00e96 Type: StoragePool
>2012-12-17 10:37:55,850 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [3484e0f5] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused
>2012-12-17 10:37:55,855 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [3484e0f5] Irs placed on server null failed. Proceed Failover
>2012-12-17 10:37:55,879 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [3484e0f5] hostFromVds::selectedVds - orchid-vds2, spmStatus Free, storage pool CL30
>2012-12-17 10:37:55,901 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [3484e0f5] SPM Init: could not find reported vds or not up - pool:CL30 vds_spm_id: 1
>2012-12-17 10:37:55,903 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [3484e0f5] SPM selection - vds seems as spm orchid-vds1
>2012-12-17 10:37:55,904 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [3484e0f5] START, SpmStopVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96), log id: 4cef7774
>2012-12-17 10:37:55,904 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [3484e0f5] SpmStopVDSCommand:: vds orchid-vds1 is in Connecting status - not performing spm stop, pool id 27ecc06e-1f3c-4470-ac7d-94740ed00e96
>2012-12-17 10:37:55,904 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [3484e0f5] FINISH, SpmStopVDSCommand, log id: 4cef7774
>2012-12-17 10:37:55,904 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [3484e0f5] spm stop on spm failed, stopping spm selection!
>2012-12-17 10:37:57,123 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-29) [221bb562] Running command: InitVdsOnUpCommand internal: true.
>2012-12-17 10:37:57,192 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-29) [58834ae] START, ValidateStorageServerConnectionVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, storageType = NFS, connectionList = [{ id: 696c5b0d-330e-4667-89c7-428563d6fc47, connection: orion.qa.lab.tlv.redhat.com:/export/meni/upgrade, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 51e4d110
>2012-12-17 10:37:57,227 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-29) [58834ae] FINISH, ValidateStorageServerConnectionVDSCommand, return: {696c5b0d-330e-4667-89c7-428563d6fc47=0}, log id: 51e4d110
>2012-12-17 10:37:57,227 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-29) [58834ae] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 27ecc06e-1f3c-4470-ac7d-94740ed00e96 Type: StoragePool
>2012-12-17 10:37:57,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-29) [58834ae] START, ConnectStorageServerVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, storageType = NFS, connectionList = [{ id: 696c5b0d-330e-4667-89c7-428563d6fc47, connection: orion.qa.lab.tlv.redhat.com:/export/meni/upgrade, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 22c5ae7d
>2012-12-17 10:37:57,250 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-29) [58834ae] FINISH, ConnectStorageServerVDSCommand, return: {696c5b0d-330e-4667-89c7-428563d6fc47=0}, log id: 22c5ae7d
>2012-12-17 10:37:57,250 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-29) [58834ae] Host orchid-vds1 storage connection was succeeded 
>2012-12-17 10:37:57,270 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-29) [58834ae] START, ConnectStoragePoolVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, vds_spm_id = 1, masterDomainId = e56c6a97-fb9a-4e87-b77b-046bd7f29be0, masterVersion = 1), log id: 7b258fe
>2012-12-17 10:38:03,949 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-29) [58834ae] FINISH, ConnectStoragePoolVDSCommand, log id: 7b258fe
>2012-12-17 10:38:03,980 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-29) [58834ae] No string for UNASSIGNED type. Use default Log
>2012-12-17 10:38:03,983 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-47) [58834ae] MultipleActionsRunner of type MigrateVmToServer invoked with no actions
>2012-12-17 10:38:04,015 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-29) [3b77614b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 7d08a922-3d3d-11e2-82f1-001122334455 Type: VDS
>2012-12-17 10:38:04,032 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-29) [dbfa1dc] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 7d08a922-3d3d-11e2-82f1-001122334455 Type: VDS
>2012-12-17 10:38:05,930 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [5ee48710] hostFromVds::selectedVds - orchid-vds1, spmStatus Free, storage pool CL30
>2012-12-17 10:38:05,955 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [5ee48710] SpmStatus on vds 7d08a922-3d3d-11e2-82f1-001122334455: Free
>2012-12-17 10:38:06,250 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [5ee48710] starting spm on vds orchid-vds1, storage pool CL30, prevId 1, LVER 1
>2012-12-17 10:38:06,252 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] START, SpmStartVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, prevId=1, prevLVER=1, storagePoolFormatType=V1, recoveryMode=Manual, SCSIFencing=false), log id: 1b7ee2fe
>2012-12-17 10:38:06,274 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] spmStart polling started: taskId = 2e0b6e5a-ce98-4b8c-81fb-92766f26e5cf
>2012-12-17 10:38:31,895 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] spmStart polling ended: taskId = 2e0b6e5a-ce98-4b8c-81fb-92766f26e5cf task status = finished
>2012-12-17 10:38:31,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] spmStart polling ended, spm status: SPM
>2012-12-17 10:38:31,916 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] START, HSMClearTaskVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, taskId=2e0b6e5a-ce98-4b8c-81fb-92766f26e5cf), log id: 7efde103
>2012-12-17 10:38:31,934 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] FINISH, HSMClearTaskVDSCommand, log id: 7efde103
>2012-12-17 10:38:31,934 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4dbfcd17, log id: 1b7ee2fe
>2012-12-17 10:38:32,193 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [5ee48710] Initialize Irs proxy from vds: orchid-vds1.qa.lab.tlv.redhat.com
>2012-12-17 10:38:32,254 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [780047c3] Attempting to get and stop tasks on storage pool CL30
>2012-12-17 10:38:32,254 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 459c68b8
>2012-12-17 10:38:32,337 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 27ecc06e-1f3c-4470-ac7d-94740ed00e96
>2012-12-17 10:38:32,339 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] START, HSMGetAllTasksInfoVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455), log id: 6d235702
>2012-12-17 10:38:32,357 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d235702
>2012-12-17 10:38:32,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 459c68b8
>2012-12-17 10:38:32,357 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [780047c3] Discovered no tasks on Storage Pool CL30
>2012-12-17 10:38:32,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2b257c48
>2012-12-17 10:38:32,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 27ecc06e-1f3c-4470-ac7d-94740ed00e96
>2012-12-17 10:38:32,359 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] START, HSMGetAllTasksInfoVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455), log id: 7440eaf0
>2012-12-17 10:38:32,376 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7440eaf0
>2012-12-17 10:38:32,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2b257c48
>2012-12-17 10:38:32,376 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [780047c3] Discovered no tasks on Storage Pool CL30
>2012-12-17 10:38:34,420 INFO  [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-3-thread-50) [12a74ae8] Running command: MigrateVmCommand internal: false. Entities affected :  ID: e9e35b3d-031d-4ef3-a8b4-4fd83936b75b Type: VM
>2012-12-17 10:38:34,432 INFO  [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-3-thread-50) [12a74ae8] START, MigrateVDSCommand(HostName = orchid-vds2, HostId = 39a7379e-439a-11e2-ba16-001122334455, vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, srcHost=orchid-vds2.qa.lab.tlv.redhat.com, dstVdsId=7d08a922-3d3d-11e2-82f1-001122334455, dstHost=orchid-vds1.qa.lab.tlv.redhat.com:54321, migrationMethod=ONLINE), log id: 460b55c6
>2012-12-17 10:38:34,433 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-3-thread-50) [12a74ae8] VdsBroker::migrate::Entered (vm_guid=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, srcHost=orchid-vds2.qa.lab.tlv.redhat.com, dstHost=orchid-vds1.qa.lab.tlv.redhat.com:54321,  method=online
>2012-12-17 10:38:34,434 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-3-thread-50) [12a74ae8] START, MigrateBrokerVDSCommand(HostName = orchid-vds2, HostId = 39a7379e-439a-11e2-ba16-001122334455, vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, srcHost=orchid-vds2.qa.lab.tlv.redhat.com, dstVdsId=7d08a922-3d3d-11e2-82f1-001122334455, dstHost=orchid-vds1.qa.lab.tlv.redhat.com:54321, migrationMethod=ONLINE), log id: 795d25b2
>2012-12-17 10:38:34,453 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-3-thread-50) [12a74ae8] FINISH, MigrateBrokerVDSCommand, log id: 795d25b2
>2012-12-17 10:38:34,487 INFO  [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-3-thread-50) [12a74ae8] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 460b55c6
>2012-12-17 10:38:34,888 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-59) [515f648d] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:38:36,951 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-61) [59abfc06] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:38:39,000 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-16) [7e43b5c5] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:38:41,121 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [5c1b0caf] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:38:43,161 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-72) [2d67b14a] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:38:45,202 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-31) [2144ca30] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:38:47,245 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-28) [3484e0f5] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:38:49,296 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-38) [7dff3777] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:38:51,405 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-50) [742a6686] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:38:53,450 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-57) [499dcbe0] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:38:55,498 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-63) [15d070bf] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:38:57,539 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-83) [2c0da98a] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:38:59,589 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) [4d8dc1c4] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:39:01,913 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-92) [565ac2be] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:39:03,958 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-58) [5cdf4906] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:39:06,003 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-48) [53480d94] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:39:08,047 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-94) [6018e3] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:39:10,098 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-62) [10ed878d] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:39:12,219 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-53) [5bc6d47] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:39:14,266 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-42) [18826afe] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:39:16,308 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-46) [60aea922] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:39:18,351 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-30) [533115ff] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:39:20,401 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-65) [77bd05d1] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done
>2012-12-17 10:39:24,000 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-4) [3f24993] START, DestroyVDSCommand(HostName = orchid-vds2, HostId = 39a7379e-439a-11e2-ba16-001122334455, vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, force=false, secondsToWait=0, gracefully=false), log id: 224d20c6
>2012-12-17 10:39:24,092 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-4) [3f24993] FINISH, DestroyVDSCommand, log id: 224d20c6
>2012-12-17 10:39:24,126 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-4) [3f24993] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b status = Down on vds orchid-vds2 ignoring it in the refresh until migration is done
>2012-12-17 10:40:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [3118271c] Checking autorecoverable hosts
>2012-12-17 10:40:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [3118271c] Autorecovering 0 hosts
>2012-12-17 10:40:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [3118271c] Checking autorecoverable hosts done
>2012-12-17 10:40:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [3118271c] Autorecovering storage domains is disabled, skipping
>2012-12-17 10:40:08,385 INFO  [org.ovirt.engine.core.bll.storage.AddNetworkCommand] (ajp-/127.0.0.1:8702-13) [12e20649] Running command: AddNetworkCommand internal: false. Entities affected :  ID: 27ecc06e-1f3c-4470-ac7d-94740ed00e96 Type: StoragePool
>2012-12-17 10:40:08,492 INFO  [org.ovirt.engine.core.bll.AttachNetworkToVdsGroupCommand] (pool-3-thread-50) [4e622b49] Running command: AttachNetworkToVdsGroupCommand internal: false. Entities affected :  ID: b3bc9dd0-4787-11e2-afa9-001122334455 Type: VdsGroups
>2012-12-17 10:40:18,663 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-50) Start time: 12/17/12 10:40 AM. Start running CanDoAction for command number 2/3 (Command type: AttachNetworkToVdsGroup)
>2012-12-17 10:40:18,666 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-47) Start time: 12/17/12 10:40 AM. Start running CanDoAction for command number 3/3 (Command type: AttachNetworkToVdsGroup)
>2012-12-17 10:40:18,670 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-48) Start time: 12/17/12 10:40 AM. Start running CanDoAction for command number 1/3 (Command type: AttachNetworkToVdsGroup)
>2012-12-17 10:40:18,678 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-47) End time: 12/17/12 10:40 AM. Finish handling CanDoAction for command number 3/3 (Command type: AttachNetworkToVdsGroup)
>2012-12-17 10:40:18,687 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-48) End time: 12/17/12 10:40 AM. Finish handling CanDoAction for command number 1/3 (Command type: AttachNetworkToVdsGroup)
>2012-12-17 10:40:18,689 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-50) End time: 12/17/12 10:40 AM. Finish handling CanDoAction for command number 2/3 (Command type: AttachNetworkToVdsGroup)
>2012-12-17 10:40:18,690 INFO  [org.ovirt.engine.core.bll.AttachNetworkToVdsGroupCommand] (pool-3-thread-50) [87ccf1e] Running command: AttachNetworkToVdsGroupCommand internal: false. Entities affected :  ID: b3bc9dd0-4787-11e2-afa9-001122334455 Type: VdsGroups
>2012-12-17 10:40:18,752 INFO  [org.ovirt.engine.core.bll.AttachNetworkToVdsGroupCommand] (pool-3-thread-50) [3a01beec] Running command: AttachNetworkToVdsGroupCommand internal: false. Entities affected :  ID: b3bc9dd0-4787-11e2-afa9-001122334455 Type: VdsGroups
>2012-12-17 10:40:18,819 INFO  [org.ovirt.engine.core.bll.AttachNetworkToVdsGroupCommand] (pool-3-thread-50) [489d95b4] Running command: AttachNetworkToVdsGroupCommand internal: false. Entities affected :  ID: b3bc9dd0-4787-11e2-afa9-001122334455 Type: VdsGroups
>2012-12-17 10:41:24,356 INFO  [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8702-2) [6d9c714c] Running command: UpdateVdsGroupCommand internal: false. Entities affected :  ID: b3bc9dd0-4787-11e2-afa9-001122334455 Type: VdsGroups
>2012-12-17 10:41:29,324 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, FullListVdsCommand( HostId = 7d08a922-3d3d-11e2-82f1-001122334455, vds=null, vmIds=[e9e35b3d-031d-4ef3-a8b4-4fd83936b75b]), log id: 3f48edde
>2012-12-17 10:41:29,344 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@4cf8aa47, log id: 3f48edde
>2012-12-17 10:41:29,893 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-44) [19b7f9b6] Recieved a memballoon Device without an address when processing VM e9e35b3d-031d-4ef3-a8b4-4fd83936b75b devices, skipping device: {specParams={model=none}, device=memballoon, type=balloon}
>2012-12-17 10:42:13,764 INFO  [org.ovirt.engine.core.bll.SetupNetworksCommand] (ajp-/127.0.0.1:8702-5) [530f614] Running command: SetupNetworksCommand internal: false. Entities affected :  ID: 7d08a922-3d3d-11e2-82f1-001122334455 Type: VDS
>2012-12-17 10:42:13,771 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp-/127.0.0.1:8702-5) [530f614] START, SetupNetworksVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, force=false, checkConnectivity=true, conectivityTimeout=120,
>	networks=[vlan_222 {id=ed425f9f-a400-46c8-8b78-625512979038, description=null, subnet=null, gateway=null, type=null, vlan_id=222, stp=false, storage_pool_id=27ecc06e-1f3c-4470-ac7d-94740ed00e96, mtu=0, vmNetwork=true, cluster=network_cluster {id={clusterId=null, networkId=null}, status=Operational, is_display=false, required=false}}],
>	bonds=[],
>	interfaces=[eth1.111 {id=cd8d98a8-ca02-4cdd-a0e2-a04884a118b8, vdsId=7d08a922-3d3d-11e2-82f1-001122334455, name=eth1.111, networkName=vlan111, vlanId=111, bootProtocol=StaticIp, address=2.2.2.2, subnet=255.255.255.255, gateway=null, mtu=9000, bridged=true, type=0, networkImplementationDetails={inSync=true, managed=true}},
>		bond0 {id=cd52dd54-1356-4381-8df5-a2cb60de8246, vdsId=7d08a922-3d3d-11e2-82f1-001122334455, name=bond0, macAddress=00:14:5E:17:D5:B0, networkName=vmnet1, bondOptions=mode=1 miimon=100, bootProtocol=StaticIp, address=1.1.1.1, subnet=255.255.255.255, gateway=null, mtu=9000, bridged=true, type=0, networkImplementationDetails={inSync=true, managed=true}},
>		eth3 {id=ffa6db0e-234b-4ebc-96aa-75b59859a649, vdsId=7d08a922-3d3d-11e2-82f1-001122334455, name=eth3, macAddress=00:14:5E:17:D5:B2, networkName=null, bondName=bond0, bootProtocol=None, address=, subnet=, gateway=null, mtu=9000, bridged=false, speed=1000, type=0, networkImplementationDetails=null},
>		eth1 {id=cdcf9e00-644e-4e2d-91b4-1d28fe0b3740, vdsId=7d08a922-3d3d-11e2-82f1-001122334455, name=eth1, macAddress=00:15:17:3D:CD:AB, networkName=null, bondName=null, bootProtocol=None, address=, subnet=, gateway=null, mtu=9000, bridged=false, speed=1000, type=0, networkImplementationDetails=null},
>		eth2 {id=d35e2d5e-3a6e-4ff7-8529-8f7c5419ebb2, vdsId=7d08a922-3d3d-11e2-82f1-001122334455, name=eth2, macAddress=00:14:5E:17:D5:B0, networkName=null, bondName=bond0, bootProtocol=None, address=, subnet=, gateway=null, mtu=9000, bridged=false, speed=1000, type=0, networkImplementationDetails=null},
>		eth0 {id=551bc1b8-a110-4d28-97fd-f6f6f4f02b1f, vdsId=7d08a922-3d3d-11e2-82f1-001122334455, name=eth0, macAddress=00:15:17:3D:CD:AA, networkName=rhevm, bondName=null, bootProtocol=Dhcp, address=10.35.97.58, subnet=255.255.255.0, gateway=10.35.97.254, mtu=1500, bridged=true, speed=1000, type=2, networkImplementationDetails={inSync=true, managed=true}},
>		eth1.222 {id=null, vdsId=7d08a922-3d3d-11e2-82f1-001122334455, macAddress=null, networkName=vlan_222, vlanId=222, bonded=null, bondName=null, bondOptions=null, bootProtocol=Dhcp, address=null, subnet=null, gateway=null, mtu=0, bridged=true, speed=null, type=0, networkImplementationDetails=null}],
>	removedNetworks=[],
>	removedBonds=[]), log id: 7a309a12
>2012-12-17 10:42:13,772 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp-/127.0.0.1:8702-5) [530f614] FINISH, SetupNetworksVDSCommand, log id: 7a309a12
>2012-12-17 10:42:16,668 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (ajp-/127.0.0.1:8702-5) [530f614] START, UpdateVdsDynamicDataVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@bd34306a), log id: 356d244c
>2012-12-17 10:42:16,669 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (ajp-/127.0.0.1:8702-5) [530f614] FINISH, UpdateVdsDynamicDataVDSCommand, log id: 356d244c
>2012-12-17 10:42:17,041 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [530f614] No string for UNASSIGNED type. Use default Log
>2012-12-17 10:42:35,556 WARN  [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-7) [4fd9154] Adding new value to configuration cache.
>2012-12-17 10:42:35,557 WARN  [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-7) [4fd9154] Didnt find the value of MaxMTU in DB for version general - using default: 9,000
>2012-12-17 10:42:35,558 WARN  [org.ovirt.engine.core.bll.storage.AddNetworkCommand] (ajp-/127.0.0.1:8702-7) [4fd9154] CanDoAction of action AddNetwork failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__NETWORK,NETWORK_MTU_OVERRIDE_NOT_SUPPORTED
>2012-12-17 10:43:09,983 WARN  [org.ovirt.engine.core.bll.storage.AddNetworkCommand] (ajp-/127.0.0.1:8702-2) [2cc9fe09] CanDoAction of action AddNetwork failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__NETWORK,NETWORK_MTU_OVERRIDE_NOT_SUPPORTED
>2012-12-17 10:45:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [1ecb2997] Checking autorecoverable hosts
>2012-12-17 10:45:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [1ecb2997] Autorecovering 0 hosts
>2012-12-17 10:45:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [1ecb2997] Checking autorecoverable hosts done
>2012-12-17 10:45:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [1ecb2997] Autorecovering storage domains is disabled, skipping
>2012-12-17 10:47:16,410 INFO  [org.ovirt.engine.core.bll.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-8) [2c86b147] Running command: AddVmInterfaceCommand internal: false. Entities affected :  ID: e9e35b3d-031d-4ef3-a8b4-4fd83936b75b Type: VM,  ID: 27ecc06e-1f3c-4470-ac7d-94740ed00e96 Type: StoragePool
>2012-12-17 10:47:16,412 INFO  [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-8) [2c86b147] MacPoolManager::allocateNewMac entered
>2012-12-17 10:47:16,412 INFO  [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-8) [2c86b147] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:16:88:59
>2012-12-17 10:47:17,473 INFO  [org.ovirt.engine.core.bll.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-8) [2f05496] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected :  ID: e9e35b3d-031d-4ef3-a8b4-4fd83936b75b Type: VM
>2012-12-17 10:47:17,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugNicVDSCommand] (ajp-/127.0.0.1:8702-8) [2f05496] START, HotPlugNicVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, vm.vm_name=vm_30, nic=VmNetworkInterface {id=c606f989-1e33-45c6-bfd1-69bd9fa38a02, networkName=vlan_222, speed=1000, type=3, name=nic4, macAddress=00:1a:4a:16:88:59, active=true, portMirroring=true, vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, vmName=vm_30, vmTemplateId=null}, vmDevice=VmDevice {vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, deviceId=c606f989-1e33-45c6-bfd1-69bd9fa38a02, device=bridge, type=interface, bootOrder=0, specParams={}, address=, managed=true, plugged=true, readOnly=false, alias=}), log id: 6bdd41b4
>2012-12-17 10:47:18,059 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugNicVDSCommand] (ajp-/127.0.0.1:8702-8) [2f05496] FINISH, HotPlugNicVDSCommand, log id: 6bdd41b4
>2012-12-17 10:47:24,064 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, FullListVdsCommand( HostId = 7d08a922-3d3d-11e2-82f1-001122334455, vds=null, vmIds=[e9e35b3d-031d-4ef3-a8b4-4fd83936b75b]), log id: 3c4dfa0f
>2012-12-17 10:47:24,083 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@235c0db7, log id: 3c4dfa0f
>2012-12-17 10:47:24,101 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-67) [1dd89be6] Recieved a memballoon Device without an address when processing VM e9e35b3d-031d-4ef3-a8b4-4fd83936b75b devices, skipping device: {specParams={model=none}, device=memballoon, type=balloon}
>2012-12-17 10:50:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [19b7f9b6] Checking autorecoverable hosts
>2012-12-17 10:50:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [19b7f9b6] Autorecovering 0 hosts
>2012-12-17 10:50:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [19b7f9b6] Checking autorecoverable hosts done
>2012-12-17 10:50:00,002 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [19b7f9b6] Autorecovering storage domains is disabled, skipping

Actions: View

Attachments on bug 887751: 664728