Bug 1372163

Summary: [RFE] Warn user about VMs that have pending snapshot removal retries
Product: Red Hat Enterprise Virtualization Manager Reporter: Germano Veit Michel <gveitmic>
Component: ovirt-engineAssignee: Ala Hino <ahino>
Status: CLOSED ERRATA QA Contact: meital avital <mavital>
Severity: urgent Docs Contact:
Priority: high    
Version: 4.0.2CC: ahino, apinnick, bcholler, fcami, fgarciad, gveitmic, gwatson, ishaby, lsurette, michal.skrivanek, mkalinin, ratamir, rbalakri, Rhev-m-bugs, srevivo, tnisan, ykaul, ylavi
Target Milestone: ovirt-4.2.0Keywords: FutureFeature, ZStream
Target Release: 4.2.0Flags: ratamir: testing_plan_complete-
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
When live or cold merge fails, snapshot disks may be left in an illegal state. If VMs with illegal snapshot disks are shut down, they will not re-start. VMs with illegal snapshot disks are now marked with an exclamation mark and a warning message not to shut them down.
Story Points: ---
Clone Of:
: 1533061 (view as bug list) Environment:
Last Closed: 2018-05-15 17:38:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1505244, 1533061    

Description Germano Veit Michel 2016-09-01 05:25:15 UTC
Description of problem:

Since latter 3.6 we have the recovery flows which can fix failed snapshot removals. Consider this case:

- A snapshot may fail to remove
- An image is left with illegal state in SD metadata
- User does not try again (ie: removal might be part of a backup script)
- Months later someone shuts down the VM

The VM will fail to come up again due to the illegal image in the SD. It's very easy to fix, but it's not obvious, and the consequence might be downtime for production VMs. A simple retry fixes the problem but this is not shown to the user.

Request:
- For VMs with failed snapshots, put a mark on the VM (something similar to the cluster upgrade icon?)
AND/OR
- Add a confirmation dialog when shutting down the VM explaining the situation

1. Proposed title of this feature request

Mark VMs that have a pending snapshot removal retry in the UI.

2. Who is the customer behind the request?

Red Hat - GSS

Comment 3 Germano Veit Michel 2016-09-01 23:33:12 UTC
See also BZ 1332038

Comment 11 Yaniv Kaul 2017-01-19 08:58:19 UTC
Tal - please assign this to someone to assess risk and complexity for the 2nd use case:
"Add a confirmation dialog when shutting down the VM explaining the situation" - I'd argue that any task should have such a warning, but we also need one for failed tasks.

Comment 12 Tal Nisan 2017-01-22 13:14:47 UTC
Idan, please have a look, we need to understand if we can gather all the info about the failed snapshot upon deactivating, passing that indication to the UI and from there add a confirmation dialog if needed

Comment 13 Idan Shaby 2017-01-23 16:15:03 UTC
We need to differentiate between two cases:
1. The active snapshot is in an illegal state - in this case, indeed the vm can't be restarted.
2. A snapshot which is not the active one (an internal snapshot) is in an illegal state - in this case, the vm *can* be restarted.

To me, in these two cases respectively, it makes sense to:
1. Add a popup when shutting down the VM explaining the situation.
2. Mark the vm in the ui (without a popup).

But this is just my opinion about this.
What do you guys think, before I assess the risk and complexity of it?

Comment 14 Marina Kalinin 2017-01-23 21:06:36 UTC
Idan,
Can you asses both please?

Comment 15 Idan Shaby 2017-01-29 12:04:48 UTC
Sure, the risk is quite low as most of the changes should be made in the ui.
We will also need to add a query to check for existing illegal snapshots in a given vm, and call it at the end of cold and live merge, and on run vm.
I guess that it will take something like ~ two weeks to complete.

Comment 18 Allon Mureinik 2017-06-29 16:11:50 UTC
I'd like to revisit the discussion here.
If removing a snapshot fails, the chain as a whole is most certianly valid, an the VM can be run - except for during a short, minimal piovt operation.
But marking the VM in some state can fail exactly like a pivot can fail, so you wouldn't be adding any resilence here, just adding another point of failure.

Can you describe a simple usecase and how this RFE will solve it?
I feel as though I might be missing something.

Comment 28 Allon Mureinik 2017-08-03 07:51:46 UTC
I see a couple of patches added here.
Just to keep the scope clear, the required patches should include
- Changes to the business entity / database / daos
- adding this new field to the search mechanism
- exposing it via the REST API

Comment 29 Ala Hino 2017-08-08 08:18:59 UTC
A note to QE guys:

As the changes to implementing this RFE includes changes to 'vms' view, please verify the performance of the system when verifying this.

Comment 30 Allon Mureinik 2017-08-08 19:02:42 UTC
(In reply to Allon Mureinik from comment #28)
> I see a couple of patches added here.
> Just to keep the scope clear, the required patches should include
> - Changes to the business entity / database / daos
Merged.

> - adding this new field to the search mechanism
Merged.

> - exposing it via the REST API
Pending review on this one. Once it's merged, the bug should be moved to MODIFIED.

Comment 31 Yaniv Lavi 2017-08-13 10:26:36 UTC
Proposed scope by engineering:
1. Cold merge should be usable as a recovery mechanism for live merge (bz#1384321 - targetted to 4.2) 
​2. Add some GUI indiciation that that the chain contains illegal imags FROM THE ENGINE'S DATABASE - a couple of days, need UXD's help.

This bug is targeting to fix #2. Is that acceptable by CEE instead of the RFE request listed in the summary?

Comment 36 Allon Mureinik 2017-09-14 12:54:23 UTC
Ala, can you add some doctext to this please?

Comment 41 Lilach Zitnitski 2017-11-14 09:15:00 UTC
--------------------------------------
Tested with the following code:
----------------------------------------
ovirt-engine-4.2.0-0.0.master.20171112130303.git8bc889c.el7.centos.noarch
vdsm-4.20.6-62.gitd3023e4.el7.centos.x86_64

Tested with the following scenario:

Steps to Reproduce:
1. Create vm with disks and start the vm
2. start live merge
3. terminate the merge and cancel the live merge 

Actual results:
An exclamation mark is shown next to the vm's name and when hovering over it a warning is shown about snapshots with illegal disks

Expected results:

Moving to VERIFIED!

Comment 46 errata-xmlrpc 2018-05-15 17:38:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:1488

Comment 47 Franta Kust 2019-05-16 13:09:42 UTC
BZ<2>Jira Resync