Bug 1533061 - [downstream clone - 4.1.9] [RFE] Warn user about VMs that have pending snapshot removal retries
Summary: [downstream clone - 4.1.9] [RFE] Warn user about VMs that have pending snapsh...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.0.2
Hardware: x86_64
OS: Linux
high
urgent
Target Milestone: ovirt-4.1.9
: ---
Assignee: Ala Hino
QA Contact: Lilach Zitnitski
URL:
Whiteboard:
Depends On: 1372163
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-10 12:03 UTC by rhev-integ
Modified: 2020-12-14 14:04 UTC (History)
21 users (show)

Fixed In Version: ovirt-engine-4.1.9.1
Doc Type: Enhancement
Doc Text:
When live or cold merge fails, snapshot disks may be left in an illegal state. If virtual machines with illegal snapshot disks are shut down, they will not re-start. Virtual machines with illegal snapshot disks are now marked with an exclamation mark and a warning message not to shut them down.
Clone Of: 1372163
Environment:
Last Closed: 2018-01-24 14:43:33 UTC
oVirt Team: Storage
Target Upstream Version:
Embargoed:
lsvaty: testing_plan_complete-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3071431 0 None None None 2018-01-10 12:11:39 UTC
Red Hat Product Errata RHBA-2018:0135 0 normal SHIPPED_LIVE org.ovirt.engine-root bug fix and enhancement update 2018-01-24 19:39:03 UTC
oVirt gerrit 80113 0 master MERGED core: Display an alert if the VM has illegal images 2020-10-15 07:49:33 UTC
oVirt gerrit 80114 0 master MERGED core: Add VM search by has_illegal_images 2020-10-15 07:49:44 UTC
oVirt gerrit 80228 0 master MERGED Add hasIllegalImages property to VM 2020-10-15 07:49:34 UTC
oVirt gerrit 80229 0 master MERGED core: Expose the VM hasIllegalImages property to REST-API 2020-10-15 07:49:34 UTC
oVirt gerrit 80638 0 master MERGED core: Enhance the VM has illegal images alert 2020-10-15 07:49:34 UTC
oVirt gerrit 86054 0 model_4.1 MERGED Add hasIllegalImages property to VM 2020-10-15 07:49:45 UTC
oVirt gerrit 86057 0 ovirt-engine-4.1 MERGED core: Expose the VM hasIllegalImages property to REST-API 2020-10-15 07:49:34 UTC

Description rhev-integ 2018-01-10 12:03:18 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1372163 +++
======================================================================

Description of problem:

Since latter 3.6 we have the recovery flows which can fix failed snapshot removals. Consider this case:

- A snapshot may fail to remove
- An image is left with illegal state in SD metadata
- User does not try again (ie: removal might be part of a backup script)
- Months later someone shuts down the VM

The VM will fail to come up again due to the illegal image in the SD. It's very easy to fix, but it's not obvious, and the consequence might be downtime for production VMs. A simple retry fixes the problem but this is not shown to the user.

Request:
- For VMs with failed snapshots, put a mark on the VM (something similar to the cluster upgrade icon?)
AND/OR
- Add a confirmation dialog when shutting down the VM explaining the situation

1. Proposed title of this feature request

Mark VMs that have a pending snapshot removal retry in the UI.

2. Who is the customer behind the request?

Red Hat - GSS

(Originally by Germano Veit Michel)

Comment 4 rhev-integ 2018-01-10 12:04:18 UTC
See also BZ 1332038

(Originally by Germano Veit Michel)

Comment 12 rhev-integ 2018-01-10 12:05:53 UTC
Tal - please assign this to someone to assess risk and complexity for the 2nd use case:
"Add a confirmation dialog when shutting down the VM explaining the situation" - I'd argue that any task should have such a warning, but we also need one for failed tasks.

(Originally by Yaniv Kaul)

Comment 13 rhev-integ 2018-01-10 12:06:01 UTC
Idan, please have a look, we need to understand if we can gather all the info about the failed snapshot upon deactivating, passing that indication to the UI and from there add a confirmation dialog if needed

(Originally by Tal Nisan)

Comment 14 rhev-integ 2018-01-10 12:06:11 UTC
We need to differentiate between two cases:
1. The active snapshot is in an illegal state - in this case, indeed the vm can't be restarted.
2. A snapshot which is not the active one (an internal snapshot) is in an illegal state - in this case, the vm *can* be restarted.

To me, in these two cases respectively, it makes sense to:
1. Add a popup when shutting down the VM explaining the situation.
2. Mark the vm in the ui (without a popup).

But this is just my opinion about this.
What do you guys think, before I assess the risk and complexity of it?

(Originally by Idan Shaby)

Comment 15 rhev-integ 2018-01-10 12:06:21 UTC
Idan,
Can you asses both please?

(Originally by Marina Kalinin)

Comment 16 rhev-integ 2018-01-10 12:06:29 UTC
Sure, the risk is quite low as most of the changes should be made in the ui.
We will also need to add a query to check for existing illegal snapshots in a given vm, and call it at the end of cold and live merge, and on run vm.
I guess that it will take something like ~ two weeks to complete.

(Originally by Idan Shaby)

Comment 19 rhev-integ 2018-01-10 12:06:57 UTC
I'd like to revisit the discussion here.
If removing a snapshot fails, the chain as a whole is most certianly valid, an the VM can be run - except for during a short, minimal piovt operation.
But marking the VM in some state can fail exactly like a pivot can fail, so you wouldn't be adding any resilence here, just adding another point of failure.

Can you describe a simple usecase and how this RFE will solve it?
I feel as though I might be missing something.

(Originally by Allon Mureinik)

Comment 29 rhev-integ 2018-01-10 12:08:37 UTC
I see a couple of patches added here.
Just to keep the scope clear, the required patches should include
- Changes to the business entity / database / daos
- adding this new field to the search mechanism
- exposing it via the REST API

(Originally by Allon Mureinik)

Comment 30 rhev-integ 2018-01-10 12:08:48 UTC
A note to QE guys:

As the changes to implementing this RFE includes changes to 'vms' view, please verify the performance of the system when verifying this.

(Originally by Ala Hino)

Comment 31 rhev-integ 2018-01-10 12:08:59 UTC
(In reply to Allon Mureinik from comment #28)
> I see a couple of patches added here.
> Just to keep the scope clear, the required patches should include
> - Changes to the business entity / database / daos
Merged.

> - adding this new field to the search mechanism
Merged.

> - exposing it via the REST API
Pending review on this one. Once it's merged, the bug should be moved to MODIFIED.

(Originally by Allon Mureinik)

Comment 32 rhev-integ 2018-01-10 12:09:11 UTC
Proposed scope by engineering:
1. Cold merge should be usable as a recovery mechanism for live merge (bz#1384321 - targetted to 4.2) 
​2. Add some GUI indiciation that that the chain contains illegal imags FROM THE ENGINE'S DATABASE - a couple of days, need UXD's help.

This bug is targeting to fix #2. Is that acceptable by CEE instead of the RFE request listed in the summary?

(Originally by ylavi)

Comment 37 rhev-integ 2018-01-10 12:10:21 UTC
Ala, can you add some doctext to this please?

(Originally by Allon Mureinik)

Comment 42 rhev-integ 2018-01-10 12:11:23 UTC
--------------------------------------
Tested with the following code:
----------------------------------------
ovirt-engine-4.2.0-0.0.master.20171112130303.git8bc889c.el7.centos.noarch
vdsm-4.20.6-62.gitd3023e4.el7.centos.x86_64

Tested with the following scenario:

Steps to Reproduce:
1. Create vm with disks and start the vm
2. start live merge
3. terminate the merge and cancel the live merge 

Actual results:
An exclamation mark is shown next to the vm's name and when hovering over it a warning is shown about snapshots with illegal disks

Expected results:

Moving to VERIFIED!

(Originally by Lilach Zitnitski)

Comment 43 Lilach Zitnitski 2018-01-17 13:17:32 UTC
--------------------------------------
Tested with the following code:
----------------------------------------
rhevm-4.1.9-0.2.el7.noarch
vdsm-4.19.44-1.el7ev.x86_64

Tested with the following scenario:

Steps to Reproduce:
1. Create vm with disks and start the vm
2. start live merge
3. terminate the merge and cancel the live merge 

Actual results:
An exclamation mark is shown next to the vm's name and when hovering over it a warning is shown about snapshots with illegal disks

Expected results:

Moving to VERIFIED!

Comment 46 errata-xmlrpc 2018-01-24 14:43:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0135

Comment 47 Franta Kust 2019-05-16 12:54:51 UTC
BZ<2>Jira re-sync


Note You need to log in before you can comment on or make changes to this bug.