Bug 1219054 - Race condition in indexing errata
Summary: Race condition in indexing errata
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Management
Version: Nightly
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: Justin Sherrill
QA Contact: sthirugn@redhat.com
URL: http://projects.theforeman.org/issues...
Whiteboard:
Depends On:
Blocks: rhci-common-installer 1212952
TreeView+ depends on / blocked
 
Reported: 2015-05-06 13:44 UTC by Walden Raines
Modified: 2017-02-23 20:04 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-08-12 14:01:38 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Foreman Issue Tracker 8586 None None None 2016-04-22 16:53:49 UTC

Description Walden Raines 2015-05-06 13:44:04 UTC
Syncing two repos at the same time which may contain the same errata can result in a race condition because find_or_create_by isn't atomic (http://www.rubydoc.info/docs/rails/ActiveRecord/Relation:find_or_create_by)


<pre>
ActiveRecord::RecordNotUnique

PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_katello_errata_on_uuid" : INSERT INTO "katello_errata" ("created_at", "description", "errata_id", "errata_type", "issued", "reboot_suggested", "severity", "solution", "summary", "title", "updated", "updated_at", "uuid") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13) RETURNING "id" 
</pre>

Comment 1 Walden Raines 2015-05-06 13:44:05 UTC
Created from redmine issue http://projects.theforeman.org/issues/8586

Comment 2 Alex Krzos 2015-05-06 13:49:11 UTC
Encountered this by Syncing RHEL 6Server RPMs and 6.6 RPMS concurrently as well as 7Server and 7.1 Server RPMS concurrently.

PGError: ERROR: duplicate key value violates unique constraint "katello_erratum_packages_eid_nvrea_n_f" DETAIL: Key (erratum_id, nvrea, name, filename)=(167, libcgroup-tools-0.41-8.el7.x86_64, libcgroup-tools, libcgroup-tools-0.41-8.el7.x86_64.rpm) already exists. : INSERT INTO "katello_erratum_packages" ("erratum_id", "filename", "name", "nvrea") VALUES ($1, $2, $3, $4) RETURNING "id"

Resuming the sync afterwards resulted in a successful completion of the sync.

Comment 4 Alex Krzos 2015-05-08 17:35:41 UTC
In some of my concurrent syncing performance tests I have now seen this issue crop up a few times.  In one instance I was concurrently syncing 10 cdn repos and three of the ten repos failed with errors:


rhel6-optional-rpms

PGError: ERROR:  duplicate key value violates unique constraint "katello_erratum_packages_eid_nvrea_n_f"
DETAIL:  Key (erratum_id, nvrea, name, filename)=(1222, xorg-x11-xinit-session-1.0.9-14.el6.x86_64, xorg-x11-xinit-session, xorg-x11-xinit-session-1.0.9-14.el6.x86_64.rpm) already exists.
: INSERT INTO "katello_erratum_packages" ("erratum_id", "filename", "name", "nvrea") VALUES ($1, $2, $3, $4) RETURNING "id"

rhel-7server-rpms

PGError: ERROR:  duplicate key value violates unique constraint "katello_erratum_packages_eid_nvrea_n_f"
DETAIL:  Key (erratum_id, nvrea, name, filename)=(1145, firefox-debuginfo-31.2.0-3.el7_0.i686, firefox-debuginfo, firefox-debuginfo-31.2.0-3.el7_0.i686.rpm) already exists.
: INSERT INTO "katello_erratum_packages" ("erratum_id", "filename", "name", "nvrea") VALUES ($1, $2, $3, $4) RETURNING "id"

rhel7-optional-rpms:

PGError: ERROR:  duplicate key value violates unique constraint "katello_erratum_packages_eid_nvrea_n_f"
DETAIL:  Key (erratum_id, nvrea, name, filename)=(334, plymouth-debuginfo-0.8.9-0.13.20140113.el7.x86_64, plymouth-debuginfo, plymouth-debuginfo-0.8.9-0.13.20140113.el7.x86_64.rpm) already exists.
: INSERT INTO "katello_erratum_packages" ("erratum_id", "filename", "name", "nvrea") VALUES ($1, $2, $3, $4) RETURNING "id"
New packages: 5756 (5.49 GB).

Comment 5 Alex Krzos 2015-05-08 17:46:33 UTC
To build on this, I have only seen this in concurrent syncs as expected with a race condition.  Out of 162 syncs (2-10 concurrent repositories x 3 tries), 12 syncs failed.  This is a ~7.5% failure rate for that sample.  The majority of the failures occurred when 7 or more repositories are synced though I have one case where this occurred when only 4 repositories were being synced concurrently.

Comment 6 Justin Sherrill 2015-05-14 14:27:29 UTC
proposing for 6.1.0 as this will bite a lot of customers

Comment 7 Og Maciel 2015-05-15 19:49:56 UTC
fwiw the workaround is to resume the dynamo task.

Comment 8 Bryan Kearney 2015-06-09 00:05:33 UTC
Moving to POST since upstream bug http://projects.theforeman.org/issues/8586 has been closed
-------------
Justin Sherrill
Applied in changeset commit:katello|d379aca4934bea62d8076b8d7772c689d7963894.

Comment 12 sthirugn@redhat.com 2015-06-16 19:02:10 UTC
I synced the following repos at the same time:
 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.3
 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.4
 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.5
 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.6

Observations:
- The sync took longer than usual.  Each repo took atleast 3 hours to complete.
- After the sync, I saw no errors in sync UI or foreman tasks
- I am able to view the errata in Content -> Errata page for each for these repos

I am marking this VERIFIED in Sat 6.1 GA Snap 8 based on the observations above.

Comment 13 Bryan Kearney 2015-08-11 13:27:21 UTC
This bug is slated to be released with Satellite 6.1.

Comment 14 Bryan Kearney 2015-08-12 14:01:38 UTC
This bug was fixed in version 6.1.1 of Satellite which was released on 12 August, 2015.


Note You need to log in before you can comment on or make changes to this bug.