Red Hat Bugzilla – Bug 1292130
[RHEV clone] Live Merge - request to remove images deleted db records without performing merge
Last modified: 2016-02-10 12:31:05 EST
+++ This bug was initially created as a clone of Bug #1280160 +++
Description of problem:
Due to a race condition, Live Merge execution can claim successful completion after only removing the snapshot record from the database, skipping the merge operation altogether.
Version-Release number of selected component (if applicable):
Witnessed on ovirt master commit 01b8febd9b; however looking through the logs it appears this is a possibility in all versions with this feature.
Based on some testing on my work laptop:
- before 3.6.0, approximately 1-in-5000 executions
- 3.6.0 or later, approximately 1-in-500 executions
(See below for more detail.)
Steps to Reproduce:
1. Execute live merge
2. Hope for the best
The best way to reproduce the bug is to make a small code change that makes reproduction much more likely. Changing the loop initialization In CommandExecutor to use TimeUnit.MILLISECONDS instead of TimeUnit.SECONDS allows reproduction after just a handful of attempts.
RemoveSnapshotCommand executes without RemoveSnapshotSingleDiskLiveCommand. This means:
- the snapshot record is removed from the db with no relevant updates
- the image record remains in the db with no relevant updates
- no vdsm verbs are called, thus the merge is not complete
- data loss after subsequent merges??
- db fields should be updated and removed as appropriate
- the merge should occur
The root cause of this is a race caused by the RemoveSnapshotCommandCallback.doPolling() method not accounting for the NOT_STARTED task status of its RemoveSnapshotSingleDiskLiveCommand children. When RemoveSnapshotCommand starts its children, the children are executed asynchronously. Usually (in my tests) the children start before the parent is complete; however, sometimes the order is reversed and this opens up a small window of time during which the callback might execute and find the child commands in the NOT_STARTED state.
When this occurs, the callback errantly considers the children to have completed successfully, causing the end-action callbacks on the parent and child commands to be triggered. The RemoveSnapshotSingleDiskCommand is thus bypassed, causing the merge to not complete.
Testing on my laptop gives evidence that the behavior is usually what is desired. 8 of 10 tests shows a favorable command sequence, whereas 2 tests showed an small (8ms) window of time where a well-timed interruption would trigger this bug. The probabilities listed above are based on this small sample.
--- Additional comment from Greg Padgett on 2015-11-11 06:27:40 IST ---
Setting target to 4.0, but this has a very small, low-risk fix that I believe should be backported.
--- Additional comment from Greg Padgett on 2015-11-11 07:19:49 IST ---
FYI this was the commit that made the problem more likely to happen in more recent releases: https://gerrit.ovirt.org/#/c/43428/ (core: capped exponential backoff for CoCo CommandExecutor, sha 3175d5a141)
--- Additional comment from Allon Mureinik on 2015-11-11 10:38:08 IST ---
Definitely should be backported to oVirt 3.6.1.
Frankly, given the triviality of the patch, I'd even backport it to the 3.5.z branch. Even though it's [much] less likely to occur there, if it does it results in a pretty nasty outcome.
included in the build.
From description it sounds like in order to reproduce/verify I need to change the code which is can't be part of verification process.
Do we have a way to verify without code change ?
The issue described here is a race condition, it is close to impossible to reproduce it without modifying the code or using a debugger.
Best course of action will be to run the live merge sanity just to make sure nothing got broken in the flow
(In reply to Tal Nisan from comment #3)
> The issue described here is a race condition, it is close to impossible to
> reproduce it without modifying the code or using a debugger.
> Best course of action will be to run the live merge sanity just to make sure
> nothing got broken in the flow
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.