Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1701898

Summary: Multiple 'WFLYWELD0039: Singleton not set for null' exceptions and errors in engine log causing tasks to be hung
Product: [oVirt] ovirt-engine Reporter: Michael Burman <mburman>
Component: BLL.InfraAssignee: Martin Perina <mperina>
Status: CLOSED CURRENTRELEASE QA Contact: Petr Matyáš <pmatyas>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.3.3.5CC: aefrat, bmaxwell, bugs, cdewolf, jstourac, lleistne, mperina, msheena, pagranat, pkremens, pmackay, reliezer, sfishbai
Target Milestone: ovirt-4.3.3-1Keywords: Automation, AutomationBlocker, Regression, Triaged
Target Release: 4.3.3.6Flags: pm-rhel: ovirt-4.3+
mperina: blocker?
lleistne: testing_ack+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: ovirt-engine-4.3.3.6 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-17 08:32:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1703792    
Attachments:
Description Flags
AddImageFromScratch_task_logs
none
15H_task_AddImageFromScratch_full_engine_logs none

Description Michael Burman 2019-04-22 12:37:35 UTC 
Description of problem:
Engine log spammed with WFLYWELD0039: Singleton not set for null errors

On latest rhv build the engine log is spammed with multiple errors:

2019-04-22 15:34:40,310+03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [] ResourceManager::refreshVdsRunTimeInfo: host=host.x.x.v.z.com(8c4c75c4-66b9-4a65-95ae-2dacfc47f469): WFLYWELD0039: Singleton not set for null. This means that you are trying to access a weld deployment with a Thread Context ClassLoader that is not associated with the deployment.


Version-Release number of selected component (if applicable):
rhvm-4.3.3.5-0.1.el7.noarch

How reproducible:
100%

Steps to Reproduce:
1. Install latest rhvm rhvm-4.3.3.5-0.1.el7.noarch

Actual results:
engine log is spammed with WFLYWELD0039: Singleton not set for null errors

Expected results:
engine log must not be spammed
also not sure what is this new error. never saw it before
Comment 1 Michael Burman 2019-04-22 12:41:41 UTC 
Seems to be related to latest jboss(not sure), this is from server.log 
2019-04-22 15:21:50,712+03 WARN  [org.jboss.as.weld] (EE-ManagedThreadFactory-engine-Thread-55761) WFLYWELD0052: Using deployment classloader to load proxy classes for module org.ovirt.engine.core.utils. Package-private access will not work. To fix this the module should declare dependencies on [org.jboss.weld.core, org.jboss.weld.spi]
Comment 3 Polina 2019-04-22 13:55:44 UTC 
these errors also appear in the last 4.2.8

ovirt-engine-4.2.8.7-0.1.el7ev.noarch
vdsm-4.20.48-1.el7ev.x86_64
libvirt-4.5.0-10.el7_6.7.x86_64
Comment 4 Shir Fishbain 2019-04-28 10:30:41 UTC 
2019-04-25 14:38:41,125+03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-43) [] ResourceManager::refreshVdsRunTimeInfo: host=host_mixed_2(fd23c20e-631b-470f-bc4b-ef6799faec44): WFLYWELD0039: Singleton not set for null. This means that you are trying to access a weld deployment with a Thread Context ClassLoader that is not associated with the deployment.

Also appears in 4.3.3.6 version 

ovirt-engine-4.3.3.6-0.1.el7.noarch
vdsm-4.30.13-1.el7ev.ppc64le
Comment 6 Avihai 2019-04-28 12:49:16 UTC 
I see this error in 4.3.3.6 on another task 'AddImageFromScratch' and it may be delaying this task as it took 15minutes(should take up to 5 minutes usually).

logs are attached.

Engine log:

Task 

2019-04-25 14:55:17,269+03 ERROR [org.ovirt.engine.core.bll.CommandsFactory] (EE-ManagedThreadFactory-engine-Thread-4668) [] An exception has occurred while trying to create a command object for command 'AddImageFromScratch' with parameters 'AddImageFromScratchParameters:{commandId='b9635c89-900b-41d0-93fe-08e0bf380695', user='null', commandType='AddImageFromScratch'}': WFLYWELD0039: Singleton not set for null. This means that you are trying to access a weld deployment with a Thread Context ClassLoader that is not associated with the deployment.
2019-04-25 14:55:17,272+03 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-4668) [] CommandAsyncTask::HandleEndActionRuntimeException: Removing CommandMultiAsyncTasks object for entity 'b9635c89-900b-41d0-93fe-08e0bf380695'


ovirt-engine-4.3.3.6-0.1.el7.noarch
vdsm-4.30.13-1.el7ev.x86_64
Comment 7 Avihai 2019-04-28 12:50:21 UTC 
Created attachment 1559613 [details]
AddImageFromScratch_task_logs
Comment 8 Avihai 2019-04-28 13:02:16 UTC 
(In reply to Avihai from comment #6)
> I see this error in 4.3.3.6 on another task 'AddImageFromScratch' and it may
> be delaying this task as it took 15minutes(should take up to 5 minutes
> usually).
> 
> logs are attached.
> 
> Engine log:
> 
> Task 
> 
> 2019-04-25 14:55:17,269+03 ERROR [org.ovirt.engine.core.bll.CommandsFactory]
> (EE-ManagedThreadFactory-engine-Thread-4668) [] An exception has occurred
> while trying to create a command object for command 'AddImageFromScratch'
> with parameters
> 'AddImageFromScratchParameters:{commandId='b9635c89-900b-41d0-93fe-
> 08e0bf380695', user='null', commandType='AddImageFromScratch'}':
> WFLYWELD0039: Singleton not set for null. This means that you are trying to
> access a weld deployment with a Thread Context ClassLoader that is not
> associated with the deployment.
> 2019-04-25 14:55:17,272+03 INFO 
> [org.ovirt.engine.core.bll.tasks.CommandAsyncTask]
> (EE-ManagedThreadFactory-engine-Thread-4668) []
> CommandAsyncTask::HandleEndActionRuntimeException: Removing
> CommandMultiAsyncTasks object for entity
> 'b9635c89-900b-41d0-93fe-08e0bf380695'
> 
> 
> ovirt-engine-4.3.3.6-0.1.el7.noarch
> vdsm-4.30.13-1.el7ev.x86_64

Correction, it's worse than I thought, this task too about 15Hours(!) to end.
This will fail all operations/tests that checks no other tasks is running like deactivating a storage domain.

Task (id= 'b9635c89-900b-41d0-93fe-08e0bf380695') start time (when adding OVF disks) 14:55:06:
> grep b9635c89-900b-41d0-93fe-08e0bf380695 engine.log                                                                                                                                              
2019-04-25 14:55:06,321+03 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-11) [6ac0cf6] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 'b9635c89-900b-41d0-93fe-08e0bf380695'

Task Error:
2019-04-25 14:55:17,269+03 ERROR [org.ovirt.engine.core.bll.CommandsFactory] (EE-ManagedThreadFactory-engine-Thread-4668) [] An exception has occurred while trying to create a command object for command 'AddImageFromScratch' with parameters 'AddImageFromScratchParameters:{commandId='b9635c89-900b-41d0-93fe-08e0bf380695', user='null', commandType='AddImageFromScratch'}': WFLYWELD0039: Singleton not set for null. This means that you are trying to access a weld deployment with a Thread Context ClassLoader that is not associated with the deployment.
2019-04-25 14:55:17,272+03 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-4668) [] CommandAsyncTask::HandleEndActionRuntimeException: Removing CommandMultiAsyncTasks object for entity 'b9635c89-900b-41d0-93fe-08e0bf380695'

In TestCase11592, Deactivate SD failed after it waited for 5 minutes(default test timeout ):
2019-04-25 15:00:09,061+03 WARN  [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainWithOvfUpdateCommand] (default task-18) [storagedomains_syncAction_894c20c8-0] Validation of action 'DeactivateStorageDomainWithOvfUpdate' failed for user admin@internal-authz. Reasons: VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ERROR_CANNOT_DEACTIVATE_DOMAIN_WITH_TASKS,$tasksNames AddImageFromScratch


Task End time 2019-04-26 05:06:43:
2019-04-26 05:06:43,315+03 INFO  [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-11) [6ac0cf6] Command 'AddDisk' id: '126ff617-af17-43a6-8957-ff5c9877525a' child commands '[b9635c89-900b-41d0-93fe-08e0bf380695]' executions were completed, status 'SUCCEEDED'

Adding full engine logs for all the run.
Comment 9 Avihai 2019-04-28 13:04:38 UTC 
Created attachment 1559615 [details]
15H_task_AddImageFromScratch_full_engine_logs
Comment 11 msheena 2019-04-29 07:57:27 UTC 
Adding another side effect here:
--------------------------------
It seems we are encountering problems deactivating storage domains (when the 'force' parameter is 'false') due to hung OVF updates.
Please notice timestamps:
  - 2019-04-25 09:41:22,238+03
  - 2019-04-25 09:41:26,927+03
  - 2019-04-25 09:41:27,937+03

2019-04-25 09:41:21,397+03 INFO  [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainWithOvfUpdateCommand] (default task-50) [storagedomains_syncAction_027b188c-f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[5b86fbc1-a104-4d24-bb7a-d41353dad800=STORAGE]', sharedLocks='[019e57ef-79ab-454a-a2d4-
41767516fc32=POOL]'}'
2019-04-25 09:41:21,457+03 INFO  [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainWithOvfUpdateCommand] (default task-50) [storagedomains_syncAction_027b188c-f] Running command: DeactivateStorageDomainWithOvfUpdateCommand internal: false. Entities affected :  ID: 5b86fbc1-a104-4d24-bb7a-d41353dad800 
Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN
2019-04-25 09:41:22,180+03 INFO  [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainWithOvfUpdateCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [storagedomains_syncAction_027b188c-f] Command 'DeactivateStorageDomainWithOvfUpdate' id 'a079dc0f-4a66-49b9-aabb-4c14922137b1' executing step 'U
PDATE_OVF_STORE'
2019-04-25 09:41:22,198+03 INFO  [org.ovirt.engine.core.bll.storage.domain.UpdateOvfStoreForStorageDomainCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [storagedomains_syncAction_027b188c-f] Running command: UpdateOvfStoreForStorageDomainCommand internal: true. Entities affected :  ID: 5b86fbc1-a104-4d
24-bb7a-d41353dad800 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN
2019-04-25 09:41:22,199+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [75ad1036] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[019e57ef-79ab-454a-a2d4-41767516fc32=OVF_UPDATE]', sharedLocks=''}'
2019-04-25 09:41:22,199+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [75ad1036] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[019e57ef-79ab-454a-a2d4-41767516fc32=OVF_UPDATE]', sharedLocks=''}'
2019-04-25 09:41:22,201+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [75ad1036] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 019e57ef-79ab-454a-a2d4-41767516fc32 Type:
 StoragePool
2019-04-25 09:41:22,209+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [75ad1036] Attempting to update VM OVFs in Data Center 'golden_env_mixed'
2019-04-25 09:41:22,215+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [75ad1036] Successfully updated VM OVFs in Data Center 'golden_env_mixed'
2019-04-25 09:41:22,215+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [75ad1036] Attempting to update template OVFs in Data Center 'golden_env_mixed'
2019-04-25 09:41:22,217+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [75ad1036] Successfully updated templates OVFs in Data Center 'golden_env_mixed'
2019-04-25 09:41:22,217+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [75ad1036] Attempting to remove unneeded template/vm OVFs in Data Center 'golden_env_mixed'
2019-04-25 09:41:22,222+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [75ad1036] Successfully removed unneeded template/vm OVFs in Data Center 'golden_env_mixed'
2019-04-25 09:41:22,223+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [75ad1036] Lock freed to object 'EngineLock:{exclusiveLocks='[019e57ef-79ab-454a-a2d4-41767516fc32=OVF_UPDATE]', sharedLocks=''}'
2019-04-25 09:41:22,235+03 INFO  [org.ovirt.engine.core.bll.exportimport.ImportVmFromConfigurationCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [storagedomains_syncAction_b25f24f2-2] Ending command 'org.ovirt.engine.core.bll.exportimport.ImportVmFromConfigurationCommand' successfully.
2019-04-25 09:41:22,236+03 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [storagedomains_syncAction_b25f24f2-2] START, SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{vmId='281d464e-e2c5-4f9d-b60c-b38001ca060b', status='Down', exitStatus='No
rmal'}), log id: 68b84d40
2019-04-25 09:41:22,236+03 ERROR [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [storagedomains_syncAction_b25f24f2-2] Command 'SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{vmId='281d464e-e2c5-4f9d-b60c-b38001ca060b', status='Down', exitStatus='
Normal'})' execution failed: null
2019-04-25 09:41:22,236+03 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [storagedomains_syncAction_b25f24f2-2] FINISH, SetVmStatusVDSCommand, return: , log id: 68b84d40
2019-04-25 09:41:22,238+03 INFO  [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [storagedomains_syncAction_b25f24f2-2] transaction rolled back
2019-04-25 09:41:22,243+03 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [storagedomains_syncAction_b25f24f2-2] Failed invoking callback end method 'onSucceeded' for command '51075136-91c9-4885-91ab-bf913c8d5613' with exception 'EngineException: ja
va.lang.NullPointerException (Failed with error ENGINE and code 5001)', the callback is marked for end method retries
2019-04-25 09:41:23,281+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[019e57ef-79ab-454a-a2d4-41767516fc32
=OVF_UPDATE]'}'
2019-04-25 09:41:23,286+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected :  ID: 5b86fbc1-a
104-4d24-bb7a-d41353dad800 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN
2019-04-25 09:41:23,296+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{storagePoolId='019e57ef-79ab-454a-a2d
4-41767516fc32', ignoreFailoverLimit='false', storageDomainId='5b86fbc1-a104-4d24-bb7a-d41353dad800', imageGroupId='e7673786-b0b0-46b9-9b0b-d6fe679ba87f', imageId='ef196f50-1f60-4fc4-aca0-1968da0eadae'}), log id: 37bb9b17
2019-04-25 09:41:23,297+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters:
2019-04-25 09:41:23,297+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ spUUID=019e57ef-79ab-454a-a2d4-41767516fc32
2019-04-25 09:41:23,297+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ sdUUID=5b86fbc1-a104-4d24-bb7a-d41353dad800
2019-04-25 09:41:23,297+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ imageGroupGUID=e7673786-b0b0-46b9-9b0b-d6fe679ba87f
2019-04-25 09:41:23,297+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ volUUID=ef196f50-1f60-4fc4-aca0-1968da0eadae
2019-04-25 09:41:23,297+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ description={"Updated":false,"Last Updated":null,"Storage Domains":[{"uuid":"5b86fbc1-a104-4d24-bb7a-d41353dad800"}
],"Disk Description":"OVF_STORE"}
2019-04-25 09:41:23,342+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] FINISH, SetVolumeDescriptionVDSCommand, return: , log id: 37bb9b17
2019-04-25 09:41:23,350+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] Lock Acquired to object 'EngineLock:{exclusiveLocks='', sharedLocks='[d7c6342b-0f6d-4ac7-8505-1073ce4de302=VDS_EXECUTION]'}'
2019-04-25 09:41:23,367+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] Running command: UploadStreamCommand internal: true. Entities affected :  ID: 5b86fbc1-a104-4d24-bb7a-d41353dad800 Type: Storage
2019-04-25 09:41:23,369+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] START, UploadStreamVDSCommand(HostName = host_mixed_2, UploadStreamVDSCommandParameters:{hostId='d7c6342b-0f6d-4ac7-8505-1073c
e4de302'}), log id: 38d9e9c
2019-04-25 09:41:23,369+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] -- executeVdsBrokerCommand, parameters:
2019-04-25 09:41:23,369+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ spUUID=019e57ef-79ab-454a-a2d4-41767516fc32
2019-04-25 09:41:23,369+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ sdUUID=5b86fbc1-a104-4d24-bb7a-d41353dad800
2019-04-25 09:41:23,369+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ imageGUID=e7673786-b0b0-46b9-9b0b-d6fe679ba87f
2019-04-25 09:41:23,369+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ volUUID=ef196f50-1f60-4fc4-aca0-1968da0eadae
2019-04-25 09:41:23,369+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ size=10240
2019-04-25 09:41:23,542+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] FINISH, UploadStreamVDSCommand, return: , log id: 38d9e9c
2019-04-25 09:41:23,554+03 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command '8d6aaa0b-0c5b-447b-8c2d-0b192d0cff8e'
2019-04-25 09:41:23,554+03 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] CommandMultiAsyncTasks::attachTask: Attaching task '15b73717-a71e-4b37-9916-803aa0e09250' to command '8d6aaa0b-0c5b-447b-8c2d-0b192d0cff8e'.
2019-04-25 09:41:23,564+03 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] Adding task '15b73717-a71e-4b37-9916-803aa0e09250' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet..
2019-04-25 09:41:23,569+03 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] BaseAsyncTask::startPollingTask: Starting to poll task '15b73717-a71e-4b37-9916-803aa0e09250'.
2019-04-25 09:41:23,572+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] Lock freed to object 'EngineLock:{exclusiveLocks='', sharedLocks='[d7c6342b-0f6d-4ac7-8505-1073ce4de302=VDS_EXECUTION]'}'
2019-04-25 09:41:23,576+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{storagePoolId='019e57ef-79ab-454a-a2d4-41767516fc32', ignoreFailoverLimit='false', storageDomainId='5b86fbc1-a104-4d24-bb7a-d41353dad800', imageGroupId='e7673786-b0b0-46b9-9b0b-d6fe679ba87f', imageId='ef196f50-1f60-4fc4-aca0-1968da0eadae'}), log id: 2f27add8
2019-04-25 09:41:23,576+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters:
2019-04-25 09:41:23,576+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ spUUID=019e57ef-79ab-454a-a2d4-41767516fc32
2019-04-25 09:41:23,576+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ sdUUID=5b86fbc1-a104-4d24-bb7a-d41353dad800
2019-04-25 09:41:23,576+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ imageGroupGUID=e7673786-b0b0-46b9-9b0b-d6fe679ba87f
2019-04-25 09:41:23,576+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ volUUID=ef196f50-1f60-4fc4-aca0-1968da0eadae
2019-04-25 09:41:23,576+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ description={"Updated":true,"Size":10240,"Last Updated":"Thu Apr 25 09:41:23 IDT 2019","Storage Domains":[{"uuid":"5b86fbc1-a104-4d24-bb7a-d41353dad800"}],"Disk Description":"OVF_STORE"}
2019-04-25 09:41:23,616+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] FINISH, SetVolumeDescriptionVDSCommand, return: , log id: 2f27add8
2019-04-25 09:41:23,619+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{storagePoolId='019e57ef-79ab-454a-a2d4-41767516fc32', ignoreFailoverLimit='false', storageDomainId='5b86fbc1-a104-4d24-bb7a-d41353dad800', imageGroupId='3953304f-4d57-4ec0-9f8a-b7b41dde3a73', imageId='4b175d42-ed60-42f8-a6f6-a1a033ef4027'}), log id: 23e9404e
2019-04-25 09:41:23,619+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters:
2019-04-25 09:41:23,620+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ spUUID=019e57ef-79ab-454a-a2d4-41767516fc32
2019-04-25 09:41:23,620+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ sdUUID=5b86fbc1-a104-4d24-bb7a-d41353dad800
2019-04-25 09:41:23,620+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ imageGroupGUID=3953304f-4d57-4ec0-9f8a-b7b41dde3a73
2019-04-25 09:41:23,620+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ volUUID=4b175d42-ed60-42f8-a6f6-a1a033ef4027
2019-04-25 09:41:23,620+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ description={"Updated":false,"Last Updated":null,"Storage Domains":[{"uuid":"5b86fbc1-a104-4d24-bb7a-d41353dad800"}],"Disk Description":"OVF_STORE"}
2019-04-25 09:41:23,656+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] FINISH, SetVolumeDescriptionVDSCommand, return: , log id: 23e9404e
2019-04-25 09:41:23,664+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] Lock Acquired to object 'EngineLock:{exclusiveLocks='', sharedLocks='[d7c6342b-0f6d-4ac7-8505-1073ce4de302=VDS_EXECUTION]'}'
2019-04-25 09:41:23,673+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] Running command: UploadStreamCommand internal: true. Entities affected :  ID: 5b86fbc1-a104-4d24-bb7a-d41353dad800 Type: Storage
2019-04-25 09:41:23,674+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] START, UploadStreamVDSCommand(HostName = host_mixed_2, UploadStreamVDSCommandParameters:{hostId='d7c6342b-0f6d-4ac7-8505-1073ce4de302'}), log id: 2e7f61ac
2019-04-25 09:41:23,674+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] -- executeVdsBrokerCommand, parameters:
2019-04-25 09:41:23,674+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ spUUID=019e57ef-79ab-454a-a2d4-41767516fc32
2019-04-25 09:41:23,674+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ sdUUID=5b86fbc1-a104-4d24-bb7a-d41353dad800
2019-04-25 09:41:23,674+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ imageGUID=3953304f-4d57-4ec0-9f8a-b7b41dde3a73
2019-04-25 09:41:23,674+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ volUUID=4b175d42-ed60-42f8-a6f6-a1a033ef4027
2019-04-25 09:41:23,674+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ size=10240
2019-04-25 09:41:23,808+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] FINISH, UploadStreamVDSCommand, return: , log id: 2e7f61ac
2019-04-25 09:41:23,819+03 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command '09cc0f89-3ee8-41c3-8734-30dc17847d22'
2019-04-25 09:41:23,819+03 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] CommandMultiAsyncTasks::attachTask: Attaching task 'f368d237-42e9-4261-b5df-201cce7c9a00' to command '09cc0f89-3ee8-41c3-8734-30dc17847d22'.
2019-04-25 09:41:23,827+03 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] Adding task 'f368d237-42e9-4261-b5df-201cce7c9a00' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet..
2019-04-25 09:41:23,831+03 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] BaseAsyncTask::startPollingTask: Starting to poll task 'f368d237-42e9-4261-b5df-201cce7c9a00'.
2019-04-25 09:41:23,834+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] Lock freed to object 'EngineLock:{exclusiveLocks='', sharedLocks='[d7c6342b-0f6d-4ac7-8505-1073ce4de302=VDS_EXECUTION]'}'
2019-04-25 09:41:23,835+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{storagePoolId='019e57ef-79ab-454a-a2d4-41767516fc32', ignoreFailoverLimit='false', storageDomainId='5b86fbc1-a104-4d24-bb7a-d41353dad800', imageGroupId='3953304f-4d57-4ec0-9f8a-b7b41dde3a73', imageId='4b175d42-ed60-42f8-a6f6-a1a033ef4027'}), log id: 28e6cead
2019-04-25 09:41:23,835+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters:
2019-04-25 09:41:23,835+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ spUUID=019e57ef-79ab-454a-a2d4-41767516fc32
2019-04-25 09:41:23,836+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ sdUUID=5b86fbc1-a104-4d24-bb7a-d41353dad800
2019-04-25 09:41:23,836+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ imageGroupGUID=3953304f-4d57-4ec0-9f8a-b7b41dde3a73
2019-04-25 09:41:23,836+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ volUUID=4b175d42-ed60-42f8-a6f6-a1a033ef4027
2019-04-25 09:41:23,836+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] ++ description={"Updated":true,"Size":10240,"Last Updated":"Thu Apr 25 09:41:23 IDT 2019","Storage Domains":[{"uuid":"5b86fbc1-a104-4d24-bb7a-d41353dad800"}],"Disk Description":"OVF_STORE"}
2019-04-25 09:41:23,878+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] FINISH, SetVolumeDescriptionVDSCommand, return: , log id: 28e6cead
2019-04-25 09:41:23,887+03 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [storagedomains_syncAction_027b188c-f] Lock freed to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[019e57ef-79ab-454a-a2d4-41767516fc32=OVF_UPDATE]'}'
2019-04-25 09:41:24,895+03 INFO  [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [storagedomains_syncAction_027b188c-f] Command 'ProcessOvfUpdateForStorageDomain' (id: '77cb2a1b-4228-4bef-bb9c-aeaaae9e767f') waiting on child command id: '8d6aaa0b-0c5b-447b-8c2d-0b192d0cff8e' type:'UploadStream' to complete
2019-04-25 09:41:24,901+03 INFO  [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [storagedomains_syncAction_027b188c-f] Command 'DeactivateStorageDomainWithOvfUpdate' (id: 'a079dc0f-4a66-49b9-aabb-4c14922137b1') waiting on child command id: '77cb2a1b-4228-4bef-bb9c-aeaaae9e767f' type:'ProcessOvfUpdateForStorageDomain' to complete
2019-04-25 09:41:25,911+03 INFO  [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-11) [storagedomains_syncAction_027b188c-f] Command 'DeactivateStorageDomainWithOvfUpdate' (id: 'a079dc0f-4a66-49b9-aabb-4c14922137b1') waiting on child command id: '77cb2a1b-4228-4bef-bb9c-aeaaae9e767f' type:'ProcessOvfUpdateForStorageDomain' to complete
2019-04-25 09:41:26,917+03 INFO  [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-8) [storagedomains_syncAction_027b188c-f] Command 'ProcessOvfUpdateForStorageDomain' (id: '77cb2a1b-4228-4bef-bb9c-aeaaae9e767f') waiting on child command id: '8d6aaa0b-0c5b-447b-8c2d-0b192d0cff8e' type:'UploadStream' to complete
2019-04-25 09:41:26,922+03 INFO  [org.ovirt.engine.core.bll.exportimport.ImportVmFromConfigurationCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-8) [storagedomains_syncAction_b25f24f2-2] Ending command 'org.ovirt.engine.core.bll.exportimport.ImportVmFromConfigurationCommand' successfully.
2019-04-25 09:41:26,923+03 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-8) [storagedomains_syncAction_b25f24f2-2] START, SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{vmId='281d464e-e2c5-4f9d-b60c-b38001ca060b', status='Down', exitStatus='Normal'}), log id: 7fa3dc90
2019-04-25 09:41:26,924+03 ERROR [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-8) [storagedomains_syncAction_b25f24f2-2] Command 'SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{vmId='281d464e-e2c5-4f9d-b60c-b38001ca060b', status='Down', exitStatus='Normal'})' execution failed: null
2019-04-25 09:41:26,924+03 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-8) [storagedomains_syncAction_b25f24f2-2] FINISH, SetVmStatusVDSCommand, return: , log id: 7fa3dc90
2019-04-25 09:41:26,927+03 INFO  [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-8) [storagedomains_syncAction_b25f24f2-2] transaction rolled back
2019-04-25 09:41:26,931+03 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-8) [storagedomains_syncAction_b25f24f2-2] Failed invoking callback end method 'onSucceeded' for command '51075136-91c9-4885-91ab-bf913c8d5613' with exception 'EngineException: java.lang.NullPointerException (Failed with error ENGINE and code 5001)', the callback is marked for end method retries

2019-04-25 09:41:27,937+03 INFO  [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [storagedomains_syncAction_027b188c-f] Command 'DeactivateStorageDomainWithOvfUpdate' (id: 'a079dc0f-4a66-49b9-aabb-4c14922137b1') waiting on child command id: '77cb2a1b-4228-4bef-bb9c-aeaaae9e767f' type:'ProcessOvfUpdateForStorageDomain' to complete
Comment 12 Michael Burman 2019-04-29 08:09:25 UTC 
This started after we upgraded to ->
eap7-wildfly-7.2.1-5.GA_redhat_00003.1.el7eap.noarch
eap7-wildfly-modules-7.2.1-5.GA_redhat_00003.1.el7eap.noarch
eap7-wildfly-java-jdk8-7.2.1-5.GA_redhat_00003.1.el7eap.noarch
Comment 15 Martin Perina 2019-05-02 08:27:38 UTC 
Moving to MODIFIED as the fix for the issue is included in EAP 7.2.1 CR3. The bug can be moved to ON_QA as soon as we have EAP 7.2.1 CR3 RPMs available
Comment 16 Martin Perina 2019-05-07 10:56:38 UTC 
Moving to ON_QE, it should be released as a part of https://errata.devel.redhat.com/advisory/41249
Comment 17 Petr Matyáš 2019-05-07 11:34:37 UTC 
Verified on ovirt-engine-4.3.3.6-0.1.el7.noarch with eap7-wildfly-7.2.1-6.GA_redhat_00004.1.el7eap.noarch