Bug 1954447

Summary: [CBT] Unable to create snapshot on a RAW disk after incremental backup
Product: [oVirt] ovirt-engine Reporter: Yury.Panchenko
Component: BLL.StorageAssignee: Eyal Shenitzky <eshenitz>
Status: CLOSED CURRENTRELEASE QA Contact: Amit Sharir <asharir>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.4.5.11CC: bugs, eshames, eshenitz, nsoffer, pchavva, sfishbai, Yury.Panchenko
Target Milestone: ovirt-4.4.7Keywords: ZStream
Target Release: 4.4.7Flags: sbonazzo: ovirt-4.4+
asharir: testing_plan_complete+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovirt-engine-4.4.7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-06 07:27:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
engine logs none

Description Yury.Panchenko 2021-04-28 07:46:12 UTC
Created attachment 1776391 [details]
engine logs

Description of problem:
image transfer fails if user deleted snapshot on a RAW disk where incremental backup was been enable

Version-Release number of selected component (if applicable):
ovirt-4.4.5
vdsm-4.40.50.10-1.el8ev.x86_64
ovirt-imageio-daemon-2.1.1-1.el8ev.x86_64
libvirt-daemon-6.6.0-13.2.module+el8.3.1+10483+85317cf0.x86_64
qemu-kvm-5.1.0-21.module+el8.3.1+10464+8ad18d1a.x86_64

How reproducible:
Create vm with QCOW and RAW disks
take a snapshot
Enable incremental backup option on both disks
Make full backup
Make incremental backup
Delete the snapshot
start incremental backup

Actual results:
backup crashed

Expected results:
RAW disk sucessfully backuped in full mode

Additional info:
2021-04-23 15:07:09,806+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-276) [e1d58a42-e58b-4e65-8e56-4e385b45d404] Failed in 'StartNbdServerVDS' method
2021-04-23 15:07:09,806+02 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-276) [e1d58a42-e58b-4e65-8e56-4e385b45d404] Unexpected return value: Status [code=2009, message=Unsupported operation: 'reason=Cannot export bitmap from RAW volume, context={}']
2021-04-23 15:07:09,815+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-276) [e1d58a42-e58b-4e65-8e56-4e385b45d404] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM pan-rhv44rel1 command StartNbdServerVDS failed: Unsupported operation: 'reason=Cannot export bitmap from RAW volume, context={}'

after unsucessfull backup engine trys to close the ticket, but can't do that

Failed to stop image transfer session. Ticket does not exist for image '4c23c18c-7804-4850-a5d1-41391fdc4482'



Unable to create snapshot on a RAW disk after incremental backup

Description of problem:
I can’t create snapshot on a VM with RAW disk after few incremental backups

Version-Release number of selected component (if applicable):


How reproducible:
ovirt-4.4.5
vdsm-4.40.50.10-1.el8ev.x86_64
ovirt-imageio-daemon-2.1.1-1.el8ev.x86_64
libvirt-daemon-6.6.0-13.2.module+el8.3.1+10483+85317cf0.x86_64
qemu-kvm-5.1.0-21.module+el8.3.1+10464+8ad18d1a.x86_64


Steps to Reproduce:
I’m not sure how reproduce that case.

1. Create vm with RAW disk
2. Take a snapshot 
3. Enable incremental backup option on the disk
4. Do few incremental backups
5. Delete snapshot. VM should be left without any snapshots and incremental backup option is disabled.
6. Try to take a new snapshot

Actual results:
Snapshot doesn’t create

Expected results:
Snapshot successfully created

Additional info:
2021-04-27 17:20:13,159+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-65) [] Failed in 'HSMGetAllTasksStatusesVDS' method
2021-04-27 17:20:13,159+02 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-65) [] Unexpected return value: TaskStatus [code=2009, message=value=Unsupported operation: "reason=Cannot add bitmaps from parent volume with raw format, context={'srcVolUUID': '52f18270-5070-4aa6-8305-8caac3248321'}" abortedcode=2009]
2021-04-27 17:20:13,177+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-65) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM pan-rhv44rel1 command HSMGetAllTasksStatusesVDS failed: value=Unsupported operation: "reason=Cannot add bitmaps from parent volume with raw format, context={'srcVolUUID': '52f18270-5070-4aa6-8305-8caac3248321'}" abortedcode=2009

Comment 1 Eyal Shenitzky 2021-04-28 08:56:04 UTC
Yury, did you try to create a live snapshot for the VM?
Does the VM run throughout that flow?

Comment 2 Yury.Panchenko 2021-04-28 09:46:08 UTC
Hello, Eyal.

> did you try to create a live snapshot for the VM?
No vm is powered off

> Does the VM run throughout that flow?
So, i started the vm and snapshot with memory sucessfully created.

Comment 3 Eyal Shenitzky 2021-04-28 09:54:17 UTC
OK, so now it is clear and we have a workaround for that issue.

Comment 4 Eyal Shenitzky 2021-04-29 07:52:20 UTC
Steps to reproduce: 
1) Create a VM with RAW format disk
2) Create a snapshot for the VM
3) Start the VM
4) Start a full backup
5) Start an incremental backup
6) Power-off the VM
7) Remove the snapshot
8) Create a new snapshot

Snapshot creation failed while it should succeed.

Comment 5 Eyal Shenitzky 2021-04-29 11:54:35 UTC
(In reply to Eyal Shenitzky from comment #4)
> Steps to reproduce: 
> 1) Create a VM with RAW format disk
> 2) Create a snapshot for the VM
> 3) Start the VM
> 4) Start a full backup
> 5) Start an incremental backup
> 6) Power-off the VM
> 7) Remove the snapshot
> 8) Create a new snapshot
> 
> Snapshot creation failed while it should succeed.

This flow also prevents from moving and copying the disk
so those should be tested also.

8.1) Create a new snapshot
8.2) Try to move the disk to a different domain
8.3) Try to copy the disk

Comment 6 Amit Sharir 2021-06-13 13:37:32 UTC
Version:
ovirt-engine-4.4.7.3-0.3.el8ev.noarch
rhv-release-4.4.7-3-001.noarch
redhat-release-8.4-0.6.el8.x86_64
redhat-release-eula-8.4-0.6.el8.x86_64
vdsm-4.40.70.3-1.el8ev.x86_64



Verification conclusions:
The expected output matched the actual output.
The total flow Eyal (steps 1-8.3) mentioned in #c5 was done with no errors/unexpected logs.
I was able to copy/move the disk as well as create additional snapshots.  


Bug verified.

Comment 7 Sandro Bonazzola 2021-07-06 07:27:58 UTC
This bugzilla is included in oVirt 4.4.7 release, published on July 6th 2021.

Since the problem described in this bug report should be resolved in oVirt 4.4.7 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.