Bug 1254003 - Deadlock in user sync causes failures in collections and aggregation- unblock 3.5 users.
Summary: Deadlock in user sync causes failures in collections and aggregation- unblock...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: oVirt
Classification: Retired
Component: ovirt-engine-dwh
Version: 3.5
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: 3.5.5
Assignee: Shirly Radco
QA Contact: Lukas Svaty
URL:
Whiteboard: dwh
Depends On:
Blocks: 1258234 1265938 1271293
TreeView+ depends on / blocked
 
Reported: 2015-08-16 13:44 UTC by Yaniv Lavi
Modified: 2016-02-10 19:28 UTC (History)
9 users (show)

Fixed In Version:
Clone Of:
: 1258234 (view as bug list)
Environment:
Last Closed: 2015-11-27 07:59:43 UTC
oVirt Team: Metrics
Embargoed:


Attachments (Terms of Use)
DWH log (4.55 MB, text/plain)
2015-08-16 13:45 UTC, Yaniv Lavi
no flags Details
Postgres Log (539.38 KB, text/plain)
2015-08-16 13:53 UTC, Yaniv Lavi
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 44884 0 master MERGED etl: removed fk that cause deadlock occasionally Never
oVirt gerrit 44885 0 ovirt-engine-dwh-3.5 MERGED etl: removed fk that cause deadlock occasionally Never

Description Yaniv Lavi 2015-08-16 13:44:04 UTC
Description of problem:
Deadlock in user sync causes failures in collections and aggregation. The users details sync is done every minute it lock the row for update, once a hour the hourly aggregation tries to sync VM data containing users foreign key which causes deadlock due to wait on the users table update.

See Thread about this issue:
http://stackoverflow.com/questions/18536147/deadlock-involving-foreign-key-constraint

and fix in 9.3 for this:
http://michael.otacoo.com/postgresql-2/postgres-9-3-feature-highlight-for-key-share-and-for-no-key-update/

Version-Release number of selected component (if applicable):
3.5.0

How reproducible:
always

Steps to Reproduce:
1. add many users to engine.
2. use users on some vm.
3. waiting a few hours for aggregation to start

Actual results:
Deadlock between aggregation (due to foreign keys) and collection.

Expected results:
Should not deadlock.

Comment 1 Yaniv Lavi 2015-08-16 13:45:01 UTC
Created attachment 1063520 [details]
DWH log

Comment 2 Yaniv Lavi 2015-08-16 13:53:16 UTC
Created attachment 1063521 [details]
Postgres Log

Comment 3 Yaniv Lavi 2015-08-16 14:42:11 UTC
To unblock 3.5 users that might hit this, we will remove the foreign key completely. For 3.6 we will try to find a better way to fix this, maybe by reducing the amount of updates done in each time.

Comment 4 Sandro Bonazzola 2015-11-27 07:59:43 UTC
Since oVirt 3.5.5 has been released, moving this from verified to closed current release.


Note You need to log in before you can comment on or make changes to this bug.