Bug 2070451
| Summary: | Deletion of multiple snapshot from the UI ( count= 10 ) ,by name : "Forklift Operator warm migration precopy" failed exit on error " Failed Removing Disks from Snapshot(s) " | ||
|---|---|---|---|
| Product: | [oVirt] ovirt-engine | Reporter: | Tzahi Ashkenazi <tashkena> |
| Component: | BLL.Storage | Assignee: | Benny Zlotnik <bzlotnik> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Shir Fishbain <sfishbai> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.4.10 | CC: | ahadas, bugs, dfodor, sfishbai |
| Target Milestone: | ovirt-4.5.1 | Flags: | pm-rhel:
ovirt-4.5?
|
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ovirt-engine-4.5.1 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-06-23 05:54:58 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | Storage | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Preliminary notes
The image that fails the lookup is 9e7bfbad-73a0-4acf-aeaf-b37449e7bbca
2022-03-31 04:24:55,041-04 INFO [org.ovirt.engine.core.bll.snapshots.RemoveDiskSnapshotsCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [7e6da9f5-0a6c-4781-9df3-e016e62d1ddd] Starting child command 5 of 10, image '9e7bfbad-73a0-4acf-aeaf-b37449e7bbca'
However, it was already removed in the previous CoCo iteration:
2022-03-31 04:24:31,146-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DestroyImageVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-13)
[7e6da9f5-0a6c-4781-9df3-e016e62d1ddd] START, DestroyImageVDSCommand( DestroyImageVDSCommandParameters:{storagePoolId='39023c53-2baf-4886-a993-3313c1828d95', ignoreFailoverLimit='false', storageDomainId='53f85c3a-2bf0-489e-8a6d-ed97a1dd25af', imageGroupId='3164cba3-4f28-4b61-8253-a0b11d898be2', imageId='00000000-0000-0000-0000-000000000000', imageList='[9e7bfbad-73a0-4acf-aeaf-b37449e7bbca]', postZero='false', force='false'}), log id: 31339238
2022-03-31 04:24:31,176-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DestroyImageVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-13) [7e6da9f5-0a6c-4781-9df3-e016e62d1ddd] FINISH, DestroyImageVDSCommand, return: , log id: 31339238
Minimal steps to reproduce: 1. Create a VM with 1 disk 2. Create 2 snapshots 3. In the UI, go to Storage Domains -> disk SD -> Disk Snapshots tab, select both snapshots and remove them Verified Both snapshots were successfully removed from UI. ovirt-engine-4.5.1.2-0.11.el8ev.noarch vdsm-4.50.1.3-1.el8ev.x86_64 This bugzilla is included in oVirt 4.5.1 release, published on June 22nd 2022. Since the problem described in this bug report should be resolved in oVirt 4.5.1 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report. |
Description of problem: Deletion of multiple snapshot from the UI from Storage > Storage Doamins > L0_Group_4_LUN3 > Disk Snapshots ( count= 10 ) ,Snapshots Description : "Forklift Operator warm migration precopy" failed exit on error " Failed Removing Disks from Snapshot(s) " those snapshot was created by MTV product > forklift-controller error from the UI "Failed Removing Disks from Snapshot(s) Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy of VM auto-rhv-red-iscsi-migration-50gb-70usage-vm-1fhgfhgfh" error from ovirt-engine log : 2022-03-31 04:23:15,641-04 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thre ad-55) [b5e53503-2e6f-4a01-8df9-93e8ba214b42] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand' with failure. 2022-03-31 04:23:15,641-04 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [b5e53503-2e6f-4a01-8df9-93e8ba214b42] Failed to transfer disk '00000000-0000-0000-0000-000000000000' (command id '1fd499fc-0f69-4b9a-8131-ad08e02c8dc0') 2022-03-31 04:24:55,046-04 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [7e6da9f5-0a6c-4781-9df3-e016e62d1ddd] Command 'RemoveDiskSnapshots' id: '0e11f3a6-0074-405c-b5ed-659af51c1c2a' with children [64e3f888-8552-4efb-a7bf-f8e2104edfe7, 45170d2f-33ff-411f-b131-da30ae0d47db, e9aaf63b-d20c-4010-81f8-d7903ccc86ca, 547f799f-49f0-4d7f-ab0f-ad4be115a722] failed when attempting to perform the next operation, marking as 'ACTIVE' 2022-03-31 04:24:55,047-04 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [7e6da9f5-0a6c-4781-9df3-e016e62d1ddd] Index 0 out of bounds for length 0: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0 at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) at java.base/java.util.Objects.checkIndex(Objects.java:372) at java.base/java.util.ArrayList.get(ArrayList.java:459) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.snapshots.RemoveDiskSnapshotsCommand.buildRemoveSnapshotSingleDiskParameters(RemoveDiskSnapshotsCommand.java:376) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.snapshots.RemoveDiskSnapshotsCommand.performNextOperationColdMerge(RemoveDiskSnapshotsCommand.java:327) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.snapshots.RemoveDiskSnapshotsCommand.performNextOperation(RemoveDiskSnapshotsCommand.java:290) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback.childCommandsExecutionEnded(SerialChildCommandsExecutionCallback.java:32) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:80) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:175) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at org.glassfish.javax.enterprise.concurrent.1.redhat-00001//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:360) at org.glassfish.javax.enterprise.concurrent.1.redhat-00001//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:511) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at org.glassfish.javax.enterprise.concurrent.1.redhat-00001//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:227) 2022-03-31 04:24:56,454-04 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveDiskSnapshotsCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [7e6da9f5-0a6c-4781-9df3-e016e62d1ddd] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveDiskSnapshotsCommand' with failure. 2022-03-31 04:24:56,538-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [7e6da9f5-0a6c-4781-9df3-e016e62d1ddd] EVENT_ID: USER_REMOVE_DISK_SNAPSHOT_FINISHED_FAILURE(376), Failed to complete deletion of Disk '1-vm-mig_Disk1' from snapshot(s) 'Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy, Forklift Operator warm migration precopy' of VM 'auto-rhv-red-iscsi-migration-50gb-70usage-vm-1' (User: admin@internal-authz). Version-Release number of selected component (if applicable): hosts: rhv-release-4.4.10-7-001.noarch engine : ovirt-engine-4.4.10.6-0.1.el8ev.noarch this bug may affect this BZ > https://bugzilla.redhat.com/show_bug.cgi?id=2053183 the engine log and vdsm logs from both active hosts related to the LUN can be found here : https://drive.google.com/drive/folders/1NLALZhFV2yGScd-mtqlZgTRMJVvFV_gc?usp=sharing