Bug 1344479

Summary:	Live Merge completes but the 'job' table entry in the database is not marked as finished
Product:	Red Hat Enterprise Virtualization Manager	Reporter:	Gordon Watson <gwatson>
Component:	ovirt-engine	Assignee:	Ala Hino <ahino>
Status:	CLOSED ERRATA	QA Contact:	Kevin Alon Goldblatt <kgoldbla>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	3.6.6	CC:	acanan, ahino, amureini, bgraveno, bmcclain, germano, gklein, gveitmic, gwatson, lsurette, mkalinin, mst, pdwyer, ratamir, rbalakri, Rhev-m-bugs, srevivo, troels, ykaul, ylavi
Target Milestone:	ovirt-4.0.0-rc2	Keywords:	ZStream
Target Release:	---	Flags:	ahino: needinfo+
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Previously, live merge tasks which completed displayed as running in the Manager. The task will now be presented as completed in the Manager, and will be removed from the appropriate database.	Story Points:	---
Clone Of:
Clones:	1348214 (view as bug list)		Environment:
Last Closed:	2016-08-23 20:42:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Storage	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1348214

Description Gordon Watson 2016-06-09 19:08:48 UTC

Description of problem:

Since upgrading to 3.6.6, we are now seeing instances of Live Merges completing, but the tasks in the Tasks pane in the RHEV Admin Portal are not being marked as having finished (an hourglass is displayed for the Finalizing step).

The 'RemoveDiskSnapshots' entry in the job table in the database is not getting changed to 'FINISHED'.

Thus it appears to the end-user that the snapshot deletion has not completed.


Version-Release number of selected component (if applicable):

RHEV 3.6.6


How reproducible:

Not, but appears to be a repeatable problem.


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 8 Allon Mureinik 2016-06-10 06:30:23 UTC

Tentatively targeting to 3.6.8 until we have an RCA.

Comment 12 Raz Tamir 2016-06-15 15:31:36 UTC

Hi Gordon,
the operation that was performed was live merge of snapshot? or snapshot's disk live deletion?

Comment 13 Gordon Watson 2016-06-15 15:42:52 UTC

It was "snapshot's disk live deletion", i.e. a single disk live snapshot deletion as opposed to an entire live snapshot deletion. Please refer to comment #6, which provides the steps on how to reproduce this. 

When I opened the bug I didn't know exactly what caused it, hence the reproduction steps were added later.

Unfortunately, I'm going to be out of the office for an extended period of time, so I hope this helps to clarify the issue.

Regards, GFW.

Comment 17 Markus Stockhausen 2016-06-17 05:26:01 UTC

*** Bug 1347243 has been marked as a duplicate of this bug. ***

Comment 18 Markus Stockhausen 2016-06-17 05:26:53 UTC

Detailed logs can be found as attachment to bug 1347243

Comment 20 Markus Stockhausen 2016-06-20 05:21:37 UTC

Hi,

I think the solution does not cover the second bug as described in BZ1347243. 

The single disk live merge command starts its action with logging the wrong message: "Disks from snapshot deletion has been completed"

See engine log of BZ1347243:

...
2016-06-16 13:21:13,630 INFO Running command: RemoveDiskSnapshotsCommand
2016-06-16 13:21:13,738 INFO deletion has been completed <- WRONG!
2016-06-16 13:21:16,444 INFO Waiting on Live Merge child commands to complete
...
2016-06-16 13:22:42,169 INGO Merge command has completed
...

Comment 21 Ala Hino 2016-06-20 06:02:46 UTC

That issue reported in BZ 1344516 and already fixed.

Comment 22 Markus Stockhausen 2016-06-20 06:04:16 UTC

Good to see. Thanks for that info.

Comment 23 Allon Mureinik 2016-06-20 11:32:41 UTC

Ala - although the fix is obvious, many customers have faced this issue. Can you please provide some doctext highlighting it?

Comment 26 Allon Mureinik 2016-06-20 13:42:23 UTC

This bug does not reproduce on the 4.0 branch (the fix is for 3.6 only), but it was cloned to follow the z-stream process.

Setting to ON_QA so QA can keep me honest here.

Comment 27 Kevin Alon Goldblatt 2016-07-18 15:18:41 UTC

Tested with the following code:
---------------------------------------
vdsm-4.18.4-2.el7ev.x86_64
rhevm-4.0.2-0.2.rc1.el7ev.noarch

Tested using the following scenario:
---------------------------------------
1. Create a VM with one disk.
2. Create a snapshot.
3. Start VM.
4. Got to main 'Storage' tab.
5. Select SD in which disk resides.
6. Go to 'Disk Snapshots' tab.
7. Select snapshot.
8. Delete snapshot.


The engine log reports that the Live Merge completed successfully
The events log in the Web Admin reports that the Live Merge completed successfully
The tasks window reports all stages of the Live Merge as having completed successfully

Moving to VERIFIED!

Comment 30 errata-xmlrpc 2016-08-23 20:42:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1743.html