Bug 1219054

Summary:	Race condition in indexing errata
Product:	Red Hat Satellite	Reporter:	Walden Raines <walden>
Component:	Content Management	Assignee:	Justin Sherrill <jsherril>
Status:	CLOSED CURRENTRELEASE	QA Contact:	sthirugn <sthirugn>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	Nightly	CC:	akrzos, bkearney, jesusr, jmatthew, jmontleo, jsherril, mmccune, omaciel, sthirugn
Target Milestone:	Unspecified
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
URL:	http://projects.theforeman.org/issues/8586
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-08-12 14:01:38 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1212602, 1212952

Description Walden Raines 2015-05-06 13:44:04 UTC

Syncing two repos at the same time which may contain the same errata can result in a race condition because find_or_create_by isn't atomic (http://www.rubydoc.info/docs/rails/ActiveRecord/Relation:find_or_create_by)


<pre>
ActiveRecord::RecordNotUnique

PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_katello_errata_on_uuid" : INSERT INTO "katello_errata" ("created_at", "description", "errata_id", "errata_type", "issued", "reboot_suggested", "severity", "solution", "summary", "title", "updated", "updated_at", "uuid") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13) RETURNING "id" 
</pre>

Comment 1 Walden Raines 2015-05-06 13:44:05 UTC

Created from redmine issue http://projects.theforeman.org/issues/8586

Comment 2 Alex Krzos 2015-05-06 13:49:11 UTC

Encountered this by Syncing RHEL 6Server RPMs and 6.6 RPMS concurrently as well as 7Server and 7.1 Server RPMS concurrently.

PGError: ERROR: duplicate key value violates unique constraint "katello_erratum_packages_eid_nvrea_n_f" DETAIL: Key (erratum_id, nvrea, name, filename)=(167, libcgroup-tools-0.41-8.el7.x86_64, libcgroup-tools, libcgroup-tools-0.41-8.el7.x86_64.rpm) already exists. : INSERT INTO "katello_erratum_packages" ("erratum_id", "filename", "name", "nvrea") VALUES ($1, $2, $3, $4) RETURNING "id"

Resuming the sync afterwards resulted in a successful completion of the sync.

Comment 4 Alex Krzos 2015-05-08 17:35:41 UTC

In some of my concurrent syncing performance tests I have now seen this issue crop up a few times.  In one instance I was concurrently syncing 10 cdn repos and three of the ten repos failed with errors:


rhel6-optional-rpms

PGError: ERROR:  duplicate key value violates unique constraint "katello_erratum_packages_eid_nvrea_n_f"
DETAIL:  Key (erratum_id, nvrea, name, filename)=(1222, xorg-x11-xinit-session-1.0.9-14.el6.x86_64, xorg-x11-xinit-session, xorg-x11-xinit-session-1.0.9-14.el6.x86_64.rpm) already exists.
: INSERT INTO "katello_erratum_packages" ("erratum_id", "filename", "name", "nvrea") VALUES ($1, $2, $3, $4) RETURNING "id"

rhel-7server-rpms

PGError: ERROR:  duplicate key value violates unique constraint "katello_erratum_packages_eid_nvrea_n_f"
DETAIL:  Key (erratum_id, nvrea, name, filename)=(1145, firefox-debuginfo-31.2.0-3.el7_0.i686, firefox-debuginfo, firefox-debuginfo-31.2.0-3.el7_0.i686.rpm) already exists.
: INSERT INTO "katello_erratum_packages" ("erratum_id", "filename", "name", "nvrea") VALUES ($1, $2, $3, $4) RETURNING "id"

rhel7-optional-rpms:

PGError: ERROR:  duplicate key value violates unique constraint "katello_erratum_packages_eid_nvrea_n_f"
DETAIL:  Key (erratum_id, nvrea, name, filename)=(334, plymouth-debuginfo-0.8.9-0.13.20140113.el7.x86_64, plymouth-debuginfo, plymouth-debuginfo-0.8.9-0.13.20140113.el7.x86_64.rpm) already exists.
: INSERT INTO "katello_erratum_packages" ("erratum_id", "filename", "name", "nvrea") VALUES ($1, $2, $3, $4) RETURNING "id"
New packages: 5756 (5.49 GB).

Comment 5 Alex Krzos 2015-05-08 17:46:33 UTC

To build on this, I have only seen this in concurrent syncs as expected with a race condition.  Out of 162 syncs (2-10 concurrent repositories x 3 tries), 12 syncs failed.  This is a ~7.5% failure rate for that sample.  The majority of the failures occurred when 7 or more repositories are synced though I have one case where this occurred when only 4 repositories were being synced concurrently.

Comment 6 Justin Sherrill 2015-05-14 14:27:29 UTC

proposing for 6.1.0 as this will bite a lot of customers

Comment 7 Og Maciel 2015-05-15 19:49:56 UTC

fwiw the workaround is to resume the dynamo task.

Comment 8 Bryan Kearney 2015-06-09 00:05:33 UTC

Moving to POST since upstream bug http://projects.theforeman.org/issues/8586 has been closed
-------------
Justin Sherrill
Applied in changeset commit:katello|d379aca4934bea62d8076b8d7772c689d7963894.

Comment 12 sthirugn@redhat.com 2015-06-16 19:02:10 UTC

I synced the following repos at the same time:
 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.3
 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.4
 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.5
 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.6

Observations:
- The sync took longer than usual.  Each repo took atleast 3 hours to complete.
- After the sync, I saw no errors in sync UI or foreman tasks
- I am able to view the errata in Content -> Errata page for each for these repos

I am marking this VERIFIED in Sat 6.1 GA Snap 8 based on the observations above.

Comment 13 Bryan Kearney 2015-08-11 13:27:21 UTC

This bug is slated to be released with Satellite 6.1.

Comment 14 Bryan Kearney 2015-08-12 14:01:38 UTC

This bug was fixed in version 6.1.1 of Satellite which was released on 12 August, 2015.