Bug 1625150

Summary: [downstream clone - 4.2.6] Creating transient disk during backup operation is failing with error "No such file or directory"
Product: Red Hat Enterprise Virtualization Manager Reporter: RHV bug bot <rhv-bugzilla-bot>
Component: ovirt-engineAssignee: Eyal Shenitzky <eshenitz>
Status: CLOSED ERRATA QA Contact: Evelina Shames <eshames>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.2.4CC: dfodor, ebenahar, eheftman, eshenitz, lsurette, michal.skrivanek, Rhev-m-bugs, srevivo, tburke, tnisan
Target Milestone: ovirt-4.2.6Keywords: ZStream
Target Release: ---Flags: lsvaty: testing_plan_complete-
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1609011 Environment:
Last Closed: 2018-09-04 13:41:47 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1609011    
Bug Blocks:    

Description RHV bug bot 2018-09-04 09:04:43 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1609011 +++
======================================================================

Description of problem:

While taking backup using Commvault (uses backup RHV api) on a VM which was down, it was observed that the creation of transient disk failed with error "No such file or directory\nCould not open backing image to determine size."

Currently attaching the snapshot disk will work even if the creation of the same snapshot is in progress. So if the "hotplugDisk" come in between the "teardownImage" process, the lv of the VM will be deactivated as a part of the teardown process. So while the time the SPM is creating the transient disk using "qemu-img create", the parent volume will be already deactivated as a part of teardown process of snapshot operation. So "qemu-img create" will fail with error "No such file or directory\nCould not open backing image to determine size."  

Version-Release number of selected component (if applicable):

rhvm-4.2.4.5-0.1.el7_3.noarch
vdsm-4.20.32-1.el7ev.x86_64

How reproducible:

Observed in the customer environment

Steps to Reproduce:

The customer is using Commvault backup to backup the VMs and issue was observed in the customer environment. However, I tried it manually using api scripts and it was not able to create this race condition.


Actual results:

The hotplugDisk operation of the snapshot is not working in some rare race condition.

Expected results:



Additional info:

(Originally by Nijin Ashok)

Comment 5 RHV bug bot 2018-09-04 09:05:08 UTC
And what is the expectation here?

(Originally by michal.skrivanek)

Comment 6 RHV bug bot 2018-09-04 09:05:12 UTC
(In reply to Michal Skrivanek from comment #3)
> And what is the expectation here?

Currently, the API allows attaching a snapshot image to another VM when the creation of the same snapshot is in progress. Do we have to prevent this?

Also, it looks like the "disk attachment" to the VM is initiated from Commvault before confirming if the snapshot creation is completed. I think this has to be changed from the Commvault end as well. I will ask the customer to raise a request to Commvault as well.

(Originally by Nijin Ashok)

Comment 8 RHV bug bot 2018-09-04 09:05:21 UTC
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops

(Originally by rhv-bugzilla-bot)

Comment 9 RHV bug bot 2018-09-04 09:05:25 UTC
Eyal, can you please provide a scenario to test this?

(Originally by Evelina Shames)

Comment 10 RHV bug bot 2018-09-04 09:05:29 UTC
Steps to reproduce:

1) Create a VM with 10 GiB disk [VM1]
2) Create another VM without disks [BACKUP_VM]
2) Create a snapshot to VM1
3) While the snapshot is created (and the VM is still locked) try to attach the 
   snapshot to BACKUP_VM using the REST-API (backup-api) 

The expected result after the fix:
The engine shouldn't allow attaching the snapshot to another VM unless the snapshot status is 'OK'

(Originally by Eyal Shenitzky)

Comment 11 RHV bug bot 2018-09-04 09:05:33 UTC
Verified.

Versions:
Engine - 4.2.6.3-1
vdsm - 4.20.37-3

(Originally by Evelina Shames)

Comment 12 RHV bug bot 2018-09-04 09:05:38 UTC
WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops

(Originally by rhv-bugzilla-bot)

Comment 13 Dusan Fodor 2018-09-04 10:28:30 UTC
moving to verified as per clone original

Comment 15 errata-xmlrpc 2018-09-04 13:41:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2623

Comment 16 Daniel Gur 2019-08-28 13:14:37 UTC
sync2jira

Comment 17 Daniel Gur 2019-08-28 13:19:39 UTC
sync2jira