Bug 1280160 - Live Merge - request to remove images deleted db records without performing merge
Live Merge - request to remove images deleted db records without performing m...
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage (Show other bugs)
Unspecified Unspecified
unspecified Severity high (vote)
: ovirt-3.5.7
: ---
Assigned To: Greg Padgett
Kevin Alon Goldblatt
Depends On:
Blocks: 1292130
  Show dependency treegraph
Reported: 2015-11-10 23:22 EST by Greg Padgett
Modified: 2016-02-10 11:55 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1292130 (view as bug list)
Last Closed: 2015-12-16 09:45:20 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
tnisan: ovirt‑3.5.z?
ylavi: planning_ack+
amureini: devel_ack+
rule-engine: testing_ack?

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 48400 master MERGED core: when starting Live Merge, check for NOT_STARTED child commands Never
oVirt gerrit 48427 ovirt-engine-3.6 MERGED core: when starting Live Merge, check for NOT_STARTED child commands Never
oVirt gerrit 48441 ovirt-engine-3.5 MERGED core: when starting Live Merge, check for NOT_STARTED child commands Never

  None (edit)
Description Greg Padgett 2015-11-10 23:22:14 EST
Description of problem:
Due to a race condition, Live Merge execution can claim successful completion after only removing the snapshot record from the database, skipping the merge operation altogether.

Version-Release number of selected component (if applicable):
Witnessed on ovirt master commit 01b8febd9b; however looking through the logs it appears this is a possibility in all versions with this feature.

How reproducible:
Based on some testing on my work laptop:
 - before 3.6.0, approximately 1-in-5000 executions
 - 3.6.0 or later, approximately 1-in-500 executions
(See below for more detail.)

Steps to Reproduce:
1. Execute live merge
2. Hope for the best
The best way to reproduce the bug is to make a small code change that makes reproduction much more likely.  Changing the loop initialization In CommandExecutor to use TimeUnit.MILLISECONDS instead of TimeUnit.SECONDS allows reproduction after just a handful of attempts.

Actual results:
RemoveSnapshotCommand executes without RemoveSnapshotSingleDiskLiveCommand.  This means:
- the snapshot record is removed from the db with no relevant updates
- the image record remains in the db with no relevant updates
- no vdsm verbs are called, thus the merge is not complete
- data loss after subsequent merges??

Expected results:
- db fields should be updated and removed as appropriate
- the merge should occur

Additional info:

The root cause of this is a race caused by the RemoveSnapshotCommandCallback.doPolling() method not accounting for the NOT_STARTED task status of its RemoveSnapshotSingleDiskLiveCommand children.  When RemoveSnapshotCommand starts its children, the children are executed asynchronously.  Usually (in my tests) the children start before the parent is complete; however, sometimes the order is reversed and this opens up a small window of time during which the callback might execute and find the child commands in the NOT_STARTED state.

When this occurs, the callback errantly considers the children to have completed successfully, causing the end-action callbacks on the parent and child commands to be triggered.  The RemoveSnapshotSingleDiskCommand is thus bypassed, causing the merge to not complete.

Testing on my laptop gives evidence that the behavior is usually what is desired.  8 of 10 tests shows a favorable command sequence, whereas 2 tests showed an small (8ms) window of time where a well-timed interruption would trigger this bug.  The probabilities listed above are based on this small sample.
Comment 1 Greg Padgett 2015-11-10 23:27:40 EST
Setting target to 4.0, but this has a very small, low-risk fix that I believe should be backported.
Comment 2 Greg Padgett 2015-11-11 00:19:49 EST
FYI this was the commit that made the problem more likely to happen in more recent releases: https://gerrit.ovirt.org/#/c/43428/ (core: capped exponential backoff for CoCo CommandExecutor, sha 3175d5a141)
Comment 3 Allon Mureinik 2015-11-11 03:38:08 EST
Definitely should be backported to oVirt 3.6.1.

Frankly, given the triviality of the patch, I'd even backport it to the 3.5.z branch. Even though it's [much] less likely to occur there, if it does it results in a pretty nasty outcome.
Comment 4 Allon Mureinik 2015-12-16 09:45:20 EST
Patch was merged to the ovirt-engine-3.5 branch, but we aren't currently planning a 3.5.7 release.

Note You need to log in before you can comment on or make changes to this bug.