Bug 1277057 - SPM commands are not sent from the manager to the hypervisor on a FIFO basis
SPM commands are not sent from the manager to the hypervisor on a FIFO basis
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
x86_64 Linux
medium Severity medium
: ovirt-4.0.0-rc
: 4.0.0
Assigned To: Liron Aravot
Carlos Mestre González
Depends On:
  Show dependency treegraph
Reported: 2015-11-02 03:44 EST by Tim Speetjens
Modified: 2017-03-21 17:28 EDT (History)
17 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-08-23 16:30:23 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 48245 master MERGED infra/storage: send commands to the SPM on a FIFO basis 2016-06-05 08:42 EDT
oVirt gerrit 58667 ovirt-engine-4.0 MERGED infra/storage: send commands to the SPM on a FIFO basis 2016-06-06 12:25 EDT

  None (edit)
Description Tim Speetjens 2015-11-02 03:44:26 EST
Description of problem:
When requests are made that need to act on the SPM, the commands are serialized on the manager (the manager sends only one SPM command at a time). The mechanism used to force this, is a synchronized object, but this does not guarantee any fair ordering of requests.

When bursts of storage commands are sent to the manager using the API, and the SPM cannot handle them quickly enough, tasks need to wait for other SPM tasks to complete. This waiting mechanism is not using a fair queue. Because of this, multiple issue may arise:
- Commands may be sent to the SPM only after severe delays, which is confusing (starvation)
- Database transactions may be aborted, if the delays are greater than the JBoss transaction timeout (at least for some commands the transaction includes waiting on this synchronized object), leading to inconsistencies in the database, requiring manual cleanup.

Version-Release number of selected component (if applicable):

How reproducible:
Using API calls/scripting and/or a slowed down filesystem
Demonstrating that the commands arrive out of order, on the SPM is easier

Steps to demonstrate commands are handled out of order:
1. Create a number of disks
2. Delete the disks one by one, but without waiting for the API command to complete (for example, using a script that uses the API to delete one disk only, and run it in the background, than launch another in the background, and so on, while leaving a small delay between the requests)
3. Compare the order of the deletes on the SPM, with the order the deletes are requested.

Actual results:
Commands arrive on the SPM out of order. Depending on the setup, maybe even transaction are aborted, and related errors are seen in the engine.log.

Expected results:
Commands should be sent from the engine to the SPM using a fair mechanism, to avoid starvation

Additional info:
Comment 1 Tim Speetjens 2015-11-02 03:57:56 EST
The use of a synchronized object is used in multiple places, and should be replaced by a  'fair' java.util.concurrent.locks.ReentrantLock, surrounding the code with a  lock() and unlock() within a try - finally block
Comment 3 Piotr Kliczewski 2015-11-04 04:12:42 EST
There was an attempt [1] to fix this issue already. Storage team decided to take over this change so changing the whiteboard.

[1] https://gerrit.ovirt.org/#/c/37947/
Comment 4 Mike McCune 2016-03-28 18:55:34 EDT
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions
Comment 5 Yaniv Lavi (Dary) 2016-05-09 06:58:13 EDT
oVirt 4.0 Alpha has been released, moving to oVirt 4.0 Beta target.
Comment 8 Allon Mureinik 2016-06-06 07:56:07 EDT
Liron, can you please backport this patch to the 4.0 branch too?
Comment 9 Tal Nisan 2016-06-06 08:25:53 EDT
I already did, Liron - please verify
Comment 10 Carlos Mestre González 2016-06-17 12:54:27 EDT
Verified this with an script, created 10 disks and then sent 10 deletes, each 0.1 seconds, checked the commands sent to the SPM from the engine and all were properly sent in the order they arrived (FIFO).

I run this test multiple times and all seemed to work.

version: rhevm-
Comment 12 errata-xmlrpc 2016-08-23 16:30:23 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.