Bug 1254003 - Deadlock in user sync causes failures in collections and aggregation- unblock 3.5 users.
Deadlock in user sync causes failures in collections and aggregation- unblock...
Status: CLOSED CURRENTRELEASE
Product: oVirt
Classification: Community
Component: ovirt-engine-dwh (Show other bugs)
3.5
x86_64 Linux
unspecified Severity high
: ---
: 3.5.5
Assigned To: Shirly Radco
Lukas Svaty
dwh
:
Depends On:
Blocks: 1258234 1265938 1271293
  Show dependency treegraph
 
Reported: 2015-08-16 09:44 EDT by Yaniv Lavi (Dary)
Modified: 2016-02-10 14:28 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1258234 (view as bug list)
Environment:
Last Closed: 2015-11-27 02:59:43 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Metrics
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
DWH log (4.55 MB, text/plain)
2015-08-16 09:45 EDT, Yaniv Lavi (Dary)
no flags Details
Postgres Log (539.38 KB, text/plain)
2015-08-16 09:53 EDT, Yaniv Lavi (Dary)
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 44884 master MERGED etl: removed fk that cause deadlock occasionally Never
oVirt gerrit 44885 ovirt-engine-dwh-3.5 MERGED etl: removed fk that cause deadlock occasionally Never

  None (edit)
Description Yaniv Lavi (Dary) 2015-08-16 09:44:04 EDT
Description of problem:
Deadlock in user sync causes failures in collections and aggregation. The users details sync is done every minute it lock the row for update, once a hour the hourly aggregation tries to sync VM data containing users foreign key which causes deadlock due to wait on the users table update.

See Thread about this issue:
http://stackoverflow.com/questions/18536147/deadlock-involving-foreign-key-constraint

and fix in 9.3 for this:
http://michael.otacoo.com/postgresql-2/postgres-9-3-feature-highlight-for-key-share-and-for-no-key-update/

Version-Release number of selected component (if applicable):
3.5.0

How reproducible:
always

Steps to Reproduce:
1. add many users to engine.
2. use users on some vm.
3. waiting a few hours for aggregation to start

Actual results:
Deadlock between aggregation (due to foreign keys) and collection.

Expected results:
Should not deadlock.
Comment 1 Yaniv Lavi (Dary) 2015-08-16 09:45:01 EDT
Created attachment 1063520 [details]
DWH log
Comment 2 Yaniv Lavi (Dary) 2015-08-16 09:53:16 EDT
Created attachment 1063521 [details]
Postgres Log
Comment 3 Yaniv Lavi (Dary) 2015-08-16 10:42:11 EDT
To unblock 3.5 users that might hit this, we will remove the foreign key completely. For 3.6 we will try to find a better way to fix this, maybe by reducing the amount of updates done in each time.
Comment 4 Sandro Bonazzola 2015-11-27 02:59:43 EST
Since oVirt 3.5.5 has been released, moving this from verified to closed current release.

Note You need to log in before you can comment on or make changes to this bug.