Bug 1280160 - Live Merge - request to remove images deleted db records without performing merge
Summary: Live Merge - request to remove images deleted db records without performing m...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage
Version: future
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Greg Padgett
QA Contact: Kevin Alon Goldblatt
URL:
Whiteboard: storage
Depends On:
Blocks: 1292130
TreeView+ depends on / blocked
 
Reported: 2015-11-11 04:22 UTC by Greg Padgett
Modified: 2021-11-12 10:05 UTC (History)
5 users (show)

Fixed In Version:
Clone Of:
: 1292130 (view as bug list)
Environment:
Last Closed: 2015-12-16 14:45:20 UTC
oVirt Team: Storage
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHV-43973 0 None None None 2021-11-12 10:05:19 UTC
oVirt gerrit 48400 0 master MERGED core: when starting Live Merge, check for NOT_STARTED child commands Never
oVirt gerrit 48427 0 ovirt-engine-3.6 MERGED core: when starting Live Merge, check for NOT_STARTED child commands Never
oVirt gerrit 48441 0 ovirt-engine-3.5 MERGED core: when starting Live Merge, check for NOT_STARTED child commands Never

Description Greg Padgett 2015-11-11 04:22:14 UTC
Description of problem:
Due to a race condition, Live Merge execution can claim successful completion after only removing the snapshot record from the database, skipping the merge operation altogether.

Version-Release number of selected component (if applicable):
Witnessed on ovirt master commit 01b8febd9b; however looking through the logs it appears this is a possibility in all versions with this feature.

How reproducible:
Based on some testing on my work laptop:
 - before 3.6.0, approximately 1-in-5000 executions
 - 3.6.0 or later, approximately 1-in-500 executions
(See below for more detail.)

Steps to Reproduce:
1. Execute live merge
2. Hope for the best
3. 
The best way to reproduce the bug is to make a small code change that makes reproduction much more likely.  Changing the loop initialization In CommandExecutor to use TimeUnit.MILLISECONDS instead of TimeUnit.SECONDS allows reproduction after just a handful of attempts.

Actual results:
RemoveSnapshotCommand executes without RemoveSnapshotSingleDiskLiveCommand.  This means:
- the snapshot record is removed from the db with no relevant updates
- the image record remains in the db with no relevant updates
- no vdsm verbs are called, thus the merge is not complete
- data loss after subsequent merges??

Expected results:
- db fields should be updated and removed as appropriate
- the merge should occur

Additional info:

The root cause of this is a race caused by the RemoveSnapshotCommandCallback.doPolling() method not accounting for the NOT_STARTED task status of its RemoveSnapshotSingleDiskLiveCommand children.  When RemoveSnapshotCommand starts its children, the children are executed asynchronously.  Usually (in my tests) the children start before the parent is complete; however, sometimes the order is reversed and this opens up a small window of time during which the callback might execute and find the child commands in the NOT_STARTED state.

When this occurs, the callback errantly considers the children to have completed successfully, causing the end-action callbacks on the parent and child commands to be triggered.  The RemoveSnapshotSingleDiskCommand is thus bypassed, causing the merge to not complete.

Testing on my laptop gives evidence that the behavior is usually what is desired.  8 of 10 tests shows a favorable command sequence, whereas 2 tests showed an small (8ms) window of time where a well-timed interruption would trigger this bug.  The probabilities listed above are based on this small sample.

Comment 1 Greg Padgett 2015-11-11 04:27:40 UTC
Setting target to 4.0, but this has a very small, low-risk fix that I believe should be backported.

Comment 2 Greg Padgett 2015-11-11 05:19:49 UTC
FYI this was the commit that made the problem more likely to happen in more recent releases: https://gerrit.ovirt.org/#/c/43428/ (core: capped exponential backoff for CoCo CommandExecutor, sha 3175d5a141)

Comment 3 Allon Mureinik 2015-11-11 08:38:08 UTC
Definitely should be backported to oVirt 3.6.1.

Frankly, given the triviality of the patch, I'd even backport it to the 3.5.z branch. Even though it's [much] less likely to occur there, if it does it results in a pretty nasty outcome.

Comment 4 Allon Mureinik 2015-12-16 14:45:20 UTC
Patch was merged to the ovirt-engine-3.5 branch, but we aren't currently planning a 3.5.7 release.
Closing.


Note You need to log in before you can comment on or make changes to this bug.