Bug 1723794 - [downstream clone - 4.3.5] Live Merge hung in the volume deletion phase, leaving snapshot in a LOCKED state
Summary: [downstream clone - 4.3.5] Live Merge hung in the volume deletion phase, lea...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.2.5
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: ovirt-4.3.5
: 4.3.5
Assignee: Eyal Shenitzky
QA Contact: Eyal Shenitzky
URL:
Whiteboard:
Depends On: 1637172
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-25 12:12 UTC by RHV bug bot
Modified: 2021-12-10 18:37 UTC (History)
8 users (show)

Fixed In Version: ovirt-engine-4.3.5.2
Doc Type: No Doc Update
Doc Text:
Clone Of: 1637172
Environment:
Last Closed: 2019-08-12 11:53:28 UTC
oVirt Team: Storage
Target Upstream Version:
Embargoed:
lsvaty: testing_plan_complete-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHV-44354 0 None None None 2021-12-10 18:37:15 UTC
Red Hat Knowledge Base (Solution) 3643122 0 None None RHV: During live merge, volume deletion hung, leaving the snapshot in a locked state 2019-06-25 12:13:34 UTC
Red Hat Product Errata RHEA-2019:2431 0 None None None 2019-08-12 11:53:40 UTC
oVirt gerrit 100818 0 'None' MERGED core: add a callback to DestroyImage command 2020-02-25 14:33:45 UTC
oVirt gerrit 100819 0 'None' ABANDONED core: limit DestroyImage retries to a 5 minutes period during merge flows 2020-02-25 14:33:43 UTC
oVirt gerrit 101052 0 'None' MERGED core: add a callback to DestroyImage command 2020-02-25 14:33:45 UTC

Description RHV bug bot 2019-06-25 12:12:47 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1637172 +++
======================================================================

Description of problem:

The volume deletion phase of a Live Merge (snapshot deletion) never completed, leaving the snapshot in a LOCKED state.

The merges for two disks completed on the host that the VM was running on. They were both active layer merges, the pivots completed. 

However, the volume deletions never completed. The engine reported a java exception for both.

During this time the SPM switched twice.


Version-Release number of selected component (if applicable):

RHV 4.2.5
RHEL 7.5 hosts w/vdsm-4.20.35-1


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

(Originally by Gordon Watson)

Comment 5 RHV bug bot 2019-06-25 12:12:57 UTC
Hey Gordon,

Did you manage to reproduce this bug?

(Originally by Eyal Shenitzky)

Comment 10 RHV bug bot 2019-06-25 12:13:05 UTC
Hey Gordon,

It is impossible to debug this log, the environment was in a huge mess and there were a lot of live merge attempts.

Please try to reproduce this issue with clear steps.

(Originally by Eyal Shenitzky)

Comment 20 RHV bug bot 2019-06-25 12:13:22 UTC
This bug has not been marked as blocker for oVirt 4.3.0.
Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1.

(Originally by Sandro Bonazzola)

Comment 22 RHV bug bot 2019-06-25 12:13:25 UTC
Reproducing this bug on a none development environment may be difficult.


Steps to reproduce:

1. Create a VM with a disk
2. Create a snapshot for the VM that contains the disk
3. Run the VM
4. Delete the snapshot from step 2
5. While the snapshot is deleted (live merge) after the 'MERGE_STATUS' step,
   block the communication between the engine to the SPM, 
   this should be done before the engine sends the SPM the command to start the deleteImage command.

The expected result after the fix:

The attempt to delete the volume will fail and a second attempt will take place.
If the communication is still blocked, the second attempt will fail too -> snapshot deletion will fail, the snapshot will not remain on 'locked'.
If the communication is OK, the second attempt will succeed -> snapshot deletion will succeed

(Originally by Eyal Shenitzky)

Comment 23 RHV bug bot 2019-06-25 12:13:27 UTC
As we discussed in thread mail and is written in the previous comment this bug is almost impossible to verify in QE(none development environment).

Eyal S. will verify this bug when it will be merged on master and a cherry-pick will be created.
 
reassigning QA Contact to Eyal to verify on current target milestone(4.3.5) .

(Originally by Avihai Efrat)

Comment 25 Avihai 2019-07-09 06:07:31 UTC
Eyal, a kind reminder to please verify this bug :)
Thank you for your help.

Comment 26 Eyal Shenitzky 2019-07-09 06:30:01 UTC
Verified locally on my dev environment by throwing an exception.

steps:
1) Create a VM with a disk
2) Create a snapshot for the VM
3) Run the VM
4) Add an Exception that will throw in -

    @Override
    protected void executeCommand() {
        getParameters().setEntityInfo(new EntityInfo(VdcObjectType.Disk, getParameters().getImageGroupId()));

        VDSReturnValue vdsReturnValue = null;
        try {
            throw new EngineException();
            // vdsReturnValue = runVdsCommand(VDSCommandType.DestroyImage, createVDSParameters());
        } catch (EngineException e) {
            log.error("Failed to delete image {}/{}", getParameters().getImageGroupId(),
                    getParameters().getImageList().stream().findFirst().get(), e);
            if (!getParameters().isLiveMerge()) {
                throw e;
            }
        }
    ....

5) Remove the created snapshot -> DestroyImageCommand failed, live merge continuing to try until it will succeed.
6) Remove the added Exception -> DestroyImageCommand succeed, snapshot removed.


Moving to verify

Comment 28 errata-xmlrpc 2019-08-12 11:53:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2431

Comment 29 Daniel Gur 2019-08-28 13:12:32 UTC
sync2jira

Comment 30 Daniel Gur 2019-08-28 13:16:44 UTC
sync2jira


Note You need to log in before you can comment on or make changes to this bug.