Created attachment 1684822 [details]
Logs

Description of problem:
One of our automation cold storage migration TCs from FC to FC failed with similar errors as in bug 1637405 (that was closed due to insufficient data)

Engine Log:
2020-05-01 07:06:03,099+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Failed in 'HSMGetAllTasksStatusesVDS' method
2020-05-01 07:06:03,109+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_1 command HSMGetAllTasksStatusesVDS failed: Error creating a new volume: (u'Volume creation 23248ce7-4bdd-4b3a-acb2-c7244fa4d057 failed: [Errno 2] No such file or directory',)
2020-05-01 07:06:03,109+03 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] SPMAsyncTask::PollTask: Polling task 'db002682-95c3-4e23-ae31-651eeb63d17b' (Parent Command 'CreateVolumeContainer', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'cleanSuccess'.
2020-05-01 07:06:03,119+03 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] BaseAsyncTask::logEndTaskFailure: Task 'db002682-95c3-4e23-ae31-651eeb63d17b' (Parent Command 'CreateVolumeContainer', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure:
-- Result: 'cleanSuccess'
-- Message: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = Error creating a new volume: (u'Volume creation 23248ce7-4bdd-4b3a-acb2-c7244fa4d057 failed: [Errno 2] No such file or directory',), code = 205',
-- Exception: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = Error creating a new volume: (u'Volume creation 23248ce7-4bdd-4b3a-acb2-c7244fa4d057 failed: [Errno 2] No such file or directory',), code = 205'
2020-05-01 07:06:03,134+03 ERROR [org.ovirt.engine.core.bll.storage.disk.image.CreateVolumeContainerCommand] (EE-ManagedThreadFactory-engine-Thread-11822) [disks_syncAction_b4c7ec70-a677-48db] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.CreateVolumeContainerCommand' with failure.
2020-05-01 07:06:06,613+03 ERROR [org.ovirt.engine.core.bll.storage.disk.image.CloneImageGroupVolumesStructureCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-81) [disks_syncAction_b4c7ec70-a677-48db] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.CloneImageGroupVolumesStructureCommand' with failure.
2020-05-01 07:06:07,633+03 ERROR [org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupWithDataCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [disks_syncAction_b4c7ec70-a677-48db] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupWithDataCommand' with failure.
2020-05-01 07:06:10,694+03 ERROR [org.ovirt.engine.core.bll.storage.disk.MoveOrCopyDiskCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-23) [disks_syncAction_b4c7ec70-a677-48db] Ending command 'org.ovirt.engine.core.bll.storage.disk.MoveOrCopyDiskCommand' with failure.
2020-05-01 07:06:10,700+03 ERROR [org.ovirt.engine.core.bll.storage.disk.image.MoveImageGroupCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-23) [disks_syncAction_b4c7ec70-a677-48db] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.MoveImageGroupCommand' with failure.
2020-05-01 07:06:10,992+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-23) [disks_syncAction_b4c7ec70-a677-48db] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: u'image=87bf99f7-fd0a-4a9c-952f-9a4a1edd5102, domain=4e85aaf2-9d13-4899-9b4e-090aa21d2c2d'
2020-05-01 07:06:10,993+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-23) [disks_syncAction_b4c7ec70-a677-48db] Command 'DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='7ccccc7d-becc-48fe-93f4-bc690d66d205', ignoreFailoverLimit='false', storageDomainId='4e85aaf2-9d13-4899-9b4e-090aa21d2c2d', imageGroupId='87bf99f7-fd0a-4a9c-952f-9a4a1edd5102', postZeros='true', discard='false', forceDelete='false'})' execution failed: IRSGenericException: IRSErrorException: Image does not exist in domain: u'image=87bf99f7-fd0a-4a9c-952f-9a4a1edd5102, domain=4e85aaf2-9d13-4899-9b4e-090aa21d2c2d'
2020-05-01 07:06:11,091+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-23) [] EVENT_ID: USER_MOVED_DISK_FINISHED_FAILURE(2,011), User admin@internal-authz have failed to move disk disk_virtioraw_0107022740 to domain fcp_1.

vdsm log:
2020-05-01 07:06:00,412+0300 ERROR (tasks/7) [storage.TaskManager.Task] (Task='db002682-95c3-4e23-ae31-651eeb63d17b') Unexpected error (task:875)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run
    return fn(*args, **kargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in run
    return self.cmd(*self.argslist, **self.argsdict)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1951, in createVolume
    initial_size=initialSize)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 930, in createVolume
    initial_size=initial_size)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 1280, in create
    (volUUID, e))
VolumeCreationError: Error creating a new volume: (u'Volume creation 23248ce7-4bdd-4b3a-acb2-c7244fa4d057 failed: [Errno 2] No such file or directory',)


Version-Release number of selected component (if applicable):
ovirt-engine-4.3.10-0.2.el7.noarch
vdsm-4.30.45-1.el7ev.x86_64

How reproducible:
Once
(Couldn't manage to reproduce again)

Steps to Reproduce:
1.Create VM From template (rhel 8.2)
2.Create and attach all disk permutations on FC
3.Move disks to another FC storage doamin

Actual results:
Moving disk virtio raw failed with the above errors.

Expected results:
Operation should succeed.

Additional info:
Logs are attached