Bug 2016701 - [CBT][Veeam] VM’ll stuck in backup state if there isn’t enough space for the scratch disk on the block storage domain
Summary: [CBT][Veeam] VM’ll stuck in backup state if there isn’t enough space for the ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage
Version: 4.4.8.5
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ovirt-4.4.9
: 4.4.9.4
Assignee: Eyal Shenitzky
QA Contact: Amit Sharir
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-10-22 19:09 UTC by Yury.Panchenko
Modified: 2021-11-20 07:48 UTC (History)
8 users (show)

Fixed In Version: ovirt-engine-4.4.9.4
Clone Of:
Environment:
Last Closed: 2021-11-20 07:48:08 UTC
oVirt Team: Storage
Embargoed:
pm-rhel: ovirt-4.4+


Attachments (Terms of Use)
engine log (347.38 KB, application/x-7z-compressed)
2021-10-22 19:09 UTC, Yury.Panchenko
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1985746 1 high CLOSED [CBT][Veeam] Full backup is stuck on 'FINALIZING' status when an error occurs during the image transfer flow 2022-08-23 11:27:19 UTC
Red Hat Issue Tracker RHV-43870 0 None None None 2021-10-22 19:12:32 UTC
oVirt gerrit 116924 0 None MERGED core: remove phase validation before stopping a backup 2021-10-26 10:19:08 UTC
oVirt gerrit 116948 0 None MERGED core: remove phase validation before stopping a backup 2021-10-26 10:14:27 UTC

Description Yury.Panchenko 2021-10-22 19:09:32 UTC
Created attachment 1836075 [details]
engine log

Description of problem:
If the block(FC or ISCSI) storage domain doesn't have enough free space for a backup operation, the vm will stuck in backup state forever

Version-Release number of selected component (if applicable):
ovirt 4.4.8.5
vdsm-4.40.80.6-1.el8ev.x86_64
ovirt-imageio-daemon-2.2.0-1.el8ev.x86_64
libvirt-daemon-7.0.0-14.3.module+el8.4.0+11878+84e54169.x86_64
qemu-kvm-5.2.0-16.module+el8.4.0+12393+838d9165.8.x86_64

How reproducible:
always

Steps to Reproduce:
1.Get a vm on a block storage domain
2.The storage domain must have less free space than the vm consumed
3.Start vm backup

Actual results:
The scratch disk hasn't been created, because not enough space error
The vm has been stucked in backup state
"The VM backup is not in READY phase, backup phase is PREPARING_SCRATCH_DISK."

Expected results:
The vm backup has been failed

Additional info:
2021-10-18 20:48:09,333+06 INFO  [org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-75) [b7d66a30-01b3-40a3-af66-f7bb01d1da14] Command 'AddDisk' (id: '62adcbfe-0410-430e-aedc-2d5169ac969b') waiting on child command id: '4a082087-742b-44f2-96ce-dd3b49eea783' type:'AddImageFromScratch' to complete
2021-10-18 20:48:09,867+06 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-61) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM node14.ecc.kz command HSMGetAllTasksStatusesVDS failed: value=Cannot create Logical Volume: 'vgname=d7ce5032-3307-4b50-bdea-c3d5c8a40e98 lvname=0667ce23-7b70-4c6c-afbb-9219649c8911 err=[\'  Volume group "d7ce5032-3307-4b50-bdea-c3d5c8a40e98" has insufficient free space (3810 extents): 3960 required.\']' abortedcode=550


2021-10-18 20:48:18,725+06 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-10) [b7d66a30-01b3-40a3-af66-f7bb01d1da14] EVENT_ID: VM_BACKUP_FAILED(10,792), Backup 71fbba21-bcbc-4f18-8a16-27a434fe5263 for VM vmthick failed (User: admin@internal-authz).

2021-10-18 20:51:08,296+06 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-10) [] Operation Failed: [Cannot stop VM backup. The VM backup is not in READY phase, backup phase is PREPARING_SCRATCH_DISK.]

Comment 1 Eyal Shenitzky 2021-10-26 10:14:27 UTC
This issue has already been fixed by - https://gerrit.ovirt.org/#/c/ovirt-engine/+/116948/.

Already available in 4.4.9.

Comment 2 Amit Sharir 2021-10-31 12:35:56 UTC
Version: 
ovirt-engine-4.4.9.4-0.1.el8ev.noarch
vdsm-4.40.90.4-1.el8ev.x86_64

Verification steps:
I did the flow mentioned by Yury.Panchenko with an ISCSI storage domain. 

Verification conclusions:
The expected output matched the actual output.
The backup failed in cases the storage domain didn't have enough free space for the backup operation. 
The VM never got stuck in a backup state (the backup simply failed).


Bug verified.

Comment 3 Sandro Bonazzola 2021-11-20 07:48:08 UTC
This bugzilla is included in  async update of oVirt engine (4.4.9.4) for oVirt 4.4.9 release, published on October 28th 2021.

Since the problem described in this bug report should be resolved in oVirt 4.4.9 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.