Description of problem: Deletion of multiple snapshot from the UI from Storage > Storage Doamins > L0_Group_4_LUN3 > Disk Snapshots ( count= 10 ) ,Snapshots Description : "Forklift Operator warm migration precopy" failed exit on error " Failed Removing Disks from Snapshot(s) " those snapshot was created by MTV product > forklift-controller error from the UI "Failed Removing Disks from Snapshot(s) Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy of VM auto-rhv-red-iscsi-migration-50gb-70usage-vm-1fhgfhgfh" error from ovirt-engine log : 2022-03-31 04:23:15,641-04 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thre ad-55) [b5e53503-2e6f-4a01-8df9-93e8ba214b42] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand' with failure. 2022-03-31 04:23:15,641-04 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [b5e53503-2e6f-4a01-8df9-93e8ba214b42] Failed to transfer disk '00000000-0000-0000-0000-000000000000' (command id '1fd499fc-0f69-4b9a-8131-ad08e02c8dc0') 2022-03-31 04:24:55,046-04 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [7e6da9f5-0a6c-4781-9df3-e016e62d1ddd] Command 'RemoveDiskSnapshots' id: '0e11f3a6-0074-405c-b5ed-659af51c1c2a' with children [64e3f888-8552-4efb-a7bf-f8e2104edfe7, 45170d2f-33ff-411f-b131-da30ae0d47db, e9aaf63b-d20c-4010-81f8-d7903ccc86ca, 547f799f-49f0-4d7f-ab0f-ad4be115a722] failed when attempting to perform the next operation, marking as 'ACTIVE' 2022-03-31 04:24:55,047-04 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [7e6da9f5-0a6c-4781-9df3-e016e62d1ddd] Index 0 out of bounds for length 0: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0 at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) at java.base/java.util.Objects.checkIndex(Objects.java:372) at java.base/java.util.ArrayList.get(ArrayList.java:459) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.snapshots.RemoveDiskSnapshotsCommand.buildRemoveSnapshotSingleDiskParameters(RemoveDiskSnapshotsCommand.java:376) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.snapshots.RemoveDiskSnapshotsCommand.performNextOperationColdMerge(RemoveDiskSnapshotsCommand.java:327) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.snapshots.RemoveDiskSnapshotsCommand.performNextOperation(RemoveDiskSnapshotsCommand.java:290) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback.childCommandsExecutionEnded(SerialChildCommandsExecutionCallback.java:32) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:80) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:175) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at org.glassfish.javax.enterprise.concurrent.1.redhat-00001//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:360) at org.glassfish.javax.enterprise.concurrent.1.redhat-00001//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:511) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at org.glassfish.javax.enterprise.concurrent.1.redhat-00001//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:227) 2022-03-31 04:24:56,454-04 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveDiskSnapshotsCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [7e6da9f5-0a6c-4781-9df3-e016e62d1ddd] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveDiskSnapshotsCommand' with failure. 2022-03-31 04:24:56,538-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [7e6da9f5-0a6c-4781-9df3-e016e62d1ddd] EVENT_ID: USER_REMOVE_DISK_SNAPSHOT_FINISHED_FAILURE(376), Failed to complete deletion of Disk '1-vm-mig_Disk1' from snapshot(s) 'Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy' of VM 'auto-rhv-red-iscsi-migration-50gb-70usage-vm-1' (User: admin@internal-authz). Version-Release number of selected component (if applicable): hosts: rhv-release-4.4.10-7-001.noarch engine : ovirt-engine-4.4.10.6-0.1.el8ev.noarch this bug may affect this BZ > https://bugzilla.redhat.com/show_bug.cgi?id=2053183 the engine log and vdsm logs from both active hosts related to the LUN can be found here : https://drive.google.com/drive/folders/1NLALZhFV2yGScd-mtqlZgTRMJVvFV_gc?usp=sharing
Preliminary notes The image that fails the lookup is 9e7bfbad-73a0-4acf-aeaf-b37449e7bbca 2022-03-31 04:24:55,041-04 INFO [org.ovirt.engine.core.bll.snapshots.RemoveDiskSnapshotsCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [7e6da9f5-0a6c-4781-9df3-e016e62d1ddd] Starting child command 5 of 10, image '9e7bfbad-73a0-4acf-aeaf-b37449e7bbca' However, it was already removed in the previous CoCo iteration: 2022-03-31 04:24:31,146-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DestroyImageVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-13) [7e6da9f5-0a6c-4781-9df3-e016e62d1ddd] START, DestroyImageVDSCommand( DestroyImageVDSCommandParameters:{storagePoolId='39023c53-2baf-4886-a993-3313c1828d95', ignoreFailoverLimit='false', storageDomainId='53f85c3a-2bf0-489e-8a6d-ed97a1dd25af', imageGroupId='3164cba3-4f28-4b61-8253-a0b11d898be2', imageId='00000000-0000-0000-0000-000000000000', imageList='[9e7bfbad-73a0-4acf-aeaf-b37449e7bbca]', postZero='false', force='false'}), log id: 31339238 2022-03-31 04:24:31,176-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DestroyImageVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-13) [7e6da9f5-0a6c-4781-9df3-e016e62d1ddd] FINISH, DestroyImageVDSCommand, return: , log id: 31339238
Minimal steps to reproduce: 1. Create a VM with 1 disk 2. Create 2 snapshots 3. In the UI, go to Storage Domains -> disk SD -> Disk Snapshots tab, select both snapshots and remove them
Verified Both snapshots were successfully removed from UI. ovirt-engine-4.5.1.2-0.11.el8ev.noarch vdsm-4.50.1.3-1.el8ev.x86_64
This bugzilla is included in oVirt 4.5.1 release, published on June 22nd 2022. Since the problem described in this bug report should be resolved in oVirt 4.5.1 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.