Created attachment 1650147 [details]
logs

Description of problem:
LSM failed in CopyDataCommand [gluster to gluster] with the following errors:
(live merge fails as well, bug 1785939]
2020-01-06 01:00:14,240+02 ERROR [org.ovirt.engine.core.bll.storage.disk.image.CopyDataCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-11) [disks_syncAction_38dba842-d05a-43dc] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.CopyDataCommand' with failure.
2020-01-06 01:00:14,246+02 INFO  [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-11) [disks_syncAction_38dba842-d05a-43dc] Command 'CopyImageGroupVolumesData' id: '00b2353e-6a7d-42c1-87a5-f80f4a860702' child commands '[15834cad-62dc-471c-9440-e880dd0f2071]' executions were completed, status 'FAILED'
2020-01-06 01:00:15,253+02 ERROR [org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupVolumesDataCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-56) [disks_syncAction_38dba842-d05a-43dc] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupVolumesDataCommand' with failure.
2020-01-06 01:00:16,328+02 INFO  [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [disks_syncAction_38dba842-d05a-43dc] Command 'LiveMigrateDisk' id: '8367eca5-a730-4926-90a2-ab5df1add758' child commands '[36eaaba5-76b5-4022-ae33-a09b5cecfcf0, c2fb29f0-09ae-47cc-8966-56dbd306f14e, 00b2353e-6a7d-42c1-87a5-f80f4a860702]' executions were completed, status 'FAILED'
2020-01-06 01:00:17,364+02 ERROR [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-57) [disks_syncAction_38dba842-d05a-43dc] Ending command 'org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand' with failure.
2020-01-06 01:00:17,365+02 ERROR [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-57) [disks_syncAction_38dba842-d05a-43dc] Failed during live storage migration of disk '5f1c154c-af37-4a81-91c5-f09d5027c06b' of vm 'a03a873b-1479-403a-ba21-7da436d303f7', attempting to end replication before deleting the target disk
2020-01-06 01:00:17,366+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-57) [disks_syncAction_38dba842-d05a-43dc] START, VmReplicateDiskFinishVDSCommand(HostName = host_mixed_1, VmReplicateDiskParameters:{hostId='6ef5c332-ea52-4a17-a3df-2f89ca0ec1fc', vmId='a03a873b-1479-403a-ba21-7da436d303f7', storagePoolId='2c770696-c781-4d21-a838-d7ef235d6152', srcStorageDomainId='8d67d096-b394-40a9-abb8-4c59fa393eb6', targetStorageDomainId='8d67d096-b394-40a9-abb8-4c59fa393eb6', imageGroupId='5f1c154c-af37-4a81-91c5-f09d5027c06b', imageId='3c9eafbd-2c9b-4436-a72a-472964646401'}), log id: 3d7c5221
2020-01-06 01:00:17,736+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-57) [disks_syncAction_38dba842-d05a-43dc] FINISH, VmReplicateDiskFinishVDSCommand, return: , log id: 3d7c5221
2020-01-06 01:00:17,736+02 ERROR [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-57) [disks_syncAction_38dba842-d05a-43dc] Attempting to delete the target of disk '5f1c154c-af37-4a81-91c5-f09d5027c06b' of vm 'a03a873b-1479-403a-ba21-7da436d303f7'
2020-01-06 01:00:18,095+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-57) [] EVENT_ID: USER_MOVED_DISK_FINISHED_FAILURE(2,011), User admin@internal-authz have failed to move disk disk_virtio_scsiraw_0600543740 to domain test_gluster_2.
020-01-06 01:00:19,310+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-3) [disks_syncAction_38dba842-d05a-43dc] Failed in 'MergeVDS' method
2020-01-06 01:00:19,317+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedExecutorService-commandCoordinator-Thread-3) [disks_syncAction_38dba842-d05a-43dc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_1 command MergeVDS failed: Merge failed
2020-01-06 01:00:19,322+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-3) [disks_syncAction_38dba842-d05a-43dc] Command 'MergeVDSCommand(HostName = host_mixed_1, MergeVDSCommandParameters:{hostId='6ef5c332-ea52-4a17-a3df-2f89ca0ec1fc', vmId='a03a873b-1479-403a-ba21-7da436d303f7', storagePoolId='2c770696-c781-4d21-a838-d7ef235d6152', storageDomainId='8d67d096-b394-40a9-abb8-4c59fa393eb6', imageGroupId='5f1c154c-af37-4a81-91c5-f09d5027c06b', imageId='3c9eafbd-2c9b-4436-a72a-472964646401', baseImageId='2ed1d699-1cd0-4f7f-9814-ba84254493d1', topImageId='3c9eafbd-2c9b-4436-a72a-472964646401', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52
2020-01-06 01:00:19,322+02 ERROR [org.ovirt.engine.core.bll.MergeCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-3) [disks_syncAction_38dba842-d05a-43dc] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 (Failed with error mergeErr and code 52)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:114)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2116)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.MergeCommand.executeCommand(MergeCommand.java:45)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1168)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1326)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2005)
        at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202)
        at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:137)
        at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1386)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:420)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:451)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:697)
        at jdk.internal.reflect.GeneratedMethodAccessor1668.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jboss.as.ee.5.GA-redhat-00002//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
        at org.jboss.invocation.1.Final-redhat-1//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.1.Final-redhat-1//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
        at org.jboss.as.weld.common.5.GA-redhat-00002//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:78)
        at org.jboss.as.weld.common.5.GA-redhat-00002//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:88)
        at org.jboss.as.weld.common.5.GA-redhat-00002//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
        at org.jboss.as.ee.5.GA-redhat-00002//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
        at org.jboss.invocation.1.Final-redhat-1//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.1.Final-redhat-1//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13)
        at jdk.internal.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566
        at org.jboss.as.ee.5.GA-redhat-00002//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
        at org.jboss.invocation.1.Final-redhat-1//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.1.Final-redhat-1//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
        at org.jboss.as.weld.common.5.GA-redhat-00002//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:78)
        at org.jboss.as.weld.common.5.GA-redhat-00002//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:88)
        at org.jboss.as.weld.common.5.GA-redhat-00002//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
        at org.jboss.as.ee.5.GA-redhat-00002//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
        at org.jboss.invocation.1.Final-redhat-1//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.1.Final-redhat-1//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13)
        at jdk.internal.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jboss.as.ee.5.GA-redhat-00002//org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89)
        at org.jboss.invocation.1.Final-redhat-1//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.5.GA-redhat-00002//org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
        at org.jboss.invocation.1.Final-redhat-1//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ee.5.GA-redhat-00002//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
        at org.jboss.invocation.1.Final-redhat-1//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.1.Final-redhat-1//org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
        at org.jboss.invocation.1.Final-redhat-1//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.1.Final-redhat-1//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
        at org.jboss.as.ee.5.GA-redhat-00002//org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
        at org.jboss.invocation.1.Final-redhat-1//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.5.GA-redhat-00002//org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53)
...



Version-Release number of selected component (if applicable):
ovirt-engine-4.4.0-0.13.master.el7.noarch
vdsm-4.40.0-164.git38a19bb.el8ev.x86_64
libvirt-5.6.0-6.module+el8.1.0+4244+9aa4e6bb.x86_64


How reproducible:
Once

Steps to Reproduce:
LSM from gluster domain to another gluster domain

Actual results:
Operation failed with the above errors

Expected results:
Operation should succeed

Additional info:
Logs are attached
---> there is a difference between engine time and hosts time of 7 hours.