Created attachment 1876367 [details] engine.log Description of problem: In hybrid mode, when trying to snapshot a VM during the finalization of the backup, it causes the backup to fail. Using SDK example - backup_vm.py, this error appears: python3 backup_vm.py -c engine incremental f261d33f-4a93-49e3-8230-94350f01bfbf --from-checkpoint-uuid 46d62e51-558d-4cf0-a8a4-3889f833edfb [ 0.0 ] Starting incremental backup for VM 'f261d33f-4a93-49e3-8230-94350f01bfbf' [ 0.6 ] Waiting until backup 'd757bbbc-638c-4c67-b055-f031f66b8065' is ready [ 18.9 ] Created checkpoint '7b05dccc-ecfa-4864-ba78-b66c6e9d79c7' [ 18.9 ] Downloading incremental backup for disk 'c90a33b3-79a3-4050-891d-eda0182a39a4' [ 18.9 ] Creating backup file './20220501164914.7b05dccc-ecfa-4864-ba78-b66c6e9d79c7.c90a33b3-79a3-4050-891d-eda0182a39a4.incremental.qcow2' [ 18.9 ] Using backing file '20220501164303.46d62e51-558d-4cf0-a8a4-3889f833edfb.c90a33b3-79a3-4050-891d-eda0182a39a4.full.qcow2' [ 20.6 ] Image transfer '8166bf39-9e38-4b05-bb7b-8ff5de1644f0' is ready [ 100.00% ] 10.00 GiB, 0.28 seconds, 35.11 GiB/s [ 20.9 ] Finalizing image transfer [ 22.9 ] Download completed successfully [ 22.9 ] Finalizing backup Traceback (most recent call last): File "backup_vm.py", line 541, in <module> main() File "backup_vm.py", line 145, in main args.command(args) File "backup_vm.py", line 182, in cmd_incremental stop_backup(connection, backup, args) File "backup_vm.py", line 370, in stop_backup backup = get_backup(connection, backup_service, backup.id) File "backup_vm.py", line 514, in get_backup last_event = get_last_backup_event(connection, backup_uuid) File "backup_vm.py", line 492, in get_last_backup_event return dict(code=event.code, description=event.description) NameError: name 'event' is not defined (Happens on both full and incremental backups) Version-Release number of selected component (if applicable): ovirt-engine-4.5.0.5-0.7.el8ev How reproducible: Always Steps to Reproduce: 1. Create a VM from rhel8.6 template and run it 2. Run backup_vm.py SDK example 3. When it starts finalizing the backup, create a live snapshot Actual results: - Backup fails: 2022-05-01 16:49:28,854+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engine ScheduledThreadPool-Thread-82) [f64b68e3-2e40-476e-9fbe-e71f4132dc9b] EVENT_ID: VM_BACKUP_FAILED(10,792), Backup d757bbbc-638c-4c67-b055-f031f66b8 065 for VM golden_env_mixed_virtio_2_0 failed (User: admin@internal-authz). - 'Auto-generated for Backup VM' snapshot remains. - The new snapshot is created successfully. Expected results: Backup should not fail in such a case Additional info: Engine logs are attached
Bumping up severity to high as it's a common use case that always reproduces on full/incremental backup, I would like QE to test it properly.
This bug seems related to VEEAM reported bug 2090682. Checkout Arik's comment https://bugzilla.redhat.com/show_bug.cgi?id=2090682#c2 .
Auto-Generated snapshot removal step of Hybrid Backup fails due to disk being locked by snapshot creation process (or any other) With this fix, in case backup succeeds, and only Auto-Generated snapshot removal during backup finalization fails - the backup is still marked as succeeded. Note that in this case Hybrid Backup command still fails, thus the user will see "Backup failed" in the event log.
(In reply to Mark Kemel from comment #4) > Auto-Generated snapshot removal step of Hybrid Backup fails due to disk > being locked by snapshot creation process (or any other) > With this fix, in case backup succeeds, and only Auto-Generated snapshot > removal during backup finalization fails - the backup is still marked as > succeeded. > > Note that in this case Hybrid Backup command still fails, thus the user will > see "Backup failed" in the event log. Verified on engine-4.5.1-0.62.el8ev Opened a new bug on having a clearer error message: bug 2096923
This bugzilla is included in oVirt 4.5.1 release, published on June 22nd 2022. Since the problem described in this bug report should be resolved in oVirt 4.5.1 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.