Bug 1344708

Summary:	Duplicate key on "katello_content_facet_errata_eid_caid" when Satellite 6.2 was put under load
Product:	Red Hat Satellite	Reporter:	Lukas Zapletal <lzap>
Component:	WebUI	Assignee:	Justin Sherrill <jsherril>
Status:	CLOSED ERRATA	QA Contact:	Roman Plevka <rplevka>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	6.2.0	CC:	alden.1, bbuckingham, bkearney, cwelton, jentrena, jhutar, jsherril, rplevka, vivpatil, zhunting
Target Milestone:	Unspecified	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
URL:	http://projects.theforeman.org/issues/15407
Whiteboard:
Fixed In Version:	tfm-rubygem-katello-3.0.0.86-1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1383481 (view as bug list)		Environment:
Last Closed:	2016-12-12 17:11:46 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1383481

Description Lukas Zapletal 2016-06-10 12:38:51 UTC

Reproducer:

Create several processes that register content hosts (subscription-manager register with an activation key with some content associated) in parallel.

2016-06-10 10:50:29 CEST ERROR:  duplicate key value violates unique constraint                      +"katello_content_facet_errata_eid_caid"
2016-06-10 10:50:29 CEST DETAIL:  Key (erratum_id, content_facet_id)=(18, 122) already exists.
2016-06-10 10:50:29 CEST STATEMENT:  INSERT INTO katello_content_facet_errata (erratum_id,           +content_facet_id) VALUES (18, 122),....
2016-06-10 10:50:29 CEST ERROR:  current transaction is aborted, commands ignored until end of       +transaction block
2016-06-10 10:50:29 CEST STATEMENT:  DELETE FROM "katello_content_facet_errata" WHERE                +"katello_content_facet_errata"."content_facet_id" = 122

2016-06-10 10:56:40 CEST ERROR:  duplicate key value violates unique constraint                      +"katello_content_facet_errata_eid_caid"
2016-06-10 10:56:40 CEST DETAIL:  Key (erratum_id, content_facet_id)=(9, 131) already exists.
2016-06-10 10:56:40 CEST STATEMENT:  INSERT INTO katello_content_facet_errata (erratum_id,           +content_facet_id) VALUES ...
2016-06-10 10:56:40 CEST ERROR:  current transaction is aborted, commands ignored until end of       +transaction block
2016-06-10 10:56:40 CEST STATEMENT:  DELETE FROM "katello_content_facet_errata" WHERE                +"katello_content_facet_errata"."content_facet_id" = 131

2016-06-10 11:05:13 CEST ERROR:  duplicate key value violates unique constraint                      +"katello_content_facet_errata_eid_caid"
2016-06-10 11:05:13 CEST DETAIL:  Key (erratum_id, content_facet_id)=(10, 149) already exists.
2016-06-10 11:05:13 CEST STATEMENT:  INSERT INTO katello_content_facet_errata (erratum_id,           +content_facet_id) VALUES ...
2016-06-10 11:05:13 CEST ERROR:  current transaction is aborted, commands ignored until end of       +transaction block
2016-06-10 11:05:13 CEST STATEMENT:  DELETE FROM "katello_content_facet_errata" WHERE                +"katello_content_facet_errata"."content_facet_id" = 149

2016-06-10 11:09:21 CEST ERROR:  duplicate key value violates unique constraint                      +"katello_content_facet_errata_eid_caid"
2016-06-10 11:09:21 CEST DETAIL:  Key (erratum_id, content_facet_id)=(1566, 147) already exists.
2016-06-10 11:09:21 CEST STATEMENT:  INSERT INTO katello_content_facet_errata (erratum_id,           +content_facet_id) VALUES (1566, 147), (2298, 147), (424, 147), (2828, 147), (2903, 147), (637,      +147), (3931, 147), (4011, 147), (4340, 147)
2016-06-10 11:09:21 CEST ERROR:  current transaction is aborted, commands ignored until end of       +transaction block
2016-06-10 11:09:21 CEST STATEMENT:  DELETE FROM "katello_content_facet_errata" WHERE                +"katello_content_facet_errata"."content_facet_id" = 147

# rpm -q satellite katello foreman
satellite-6.2.0-14.2.el7sat.noarch
katello-3.0.0-6.el7sat.noarch
foreman-1.11.0.34-1.el7sat.noarch

Debug tarball (Foreman core team permissions needed to download it):
http://debugs.theforeman.org/foreman-debug-0Qsen.tar.xz

I am able to randomly reproduce on my own box, takes hours to hit with about 100 clients registering.

Comment 1 Lukas Zapletal 2016-06-10 12:44:43 UTC

Note: You may need to configure PassengerMinInstances 2 in /etc/httpd/conf.d/05-foreman-ssl.conf in order to have at least two processes that interfere each other.

Comment 5 Justin Sherrill 2016-06-14 12:45:54 UTC

Created redmine issue http://projects.theforeman.org/issues/15407 from this bug

Comment 6 Bryan Kearney 2016-06-14 14:17:22 UTC

Upstream bug component is WebUI

Comment 7 Lukas Zapletal 2016-06-20 12:32:54 UTC

WORKAROUND:

cat >>/etc/httpd/conf.d/passenger.conf <<EOPS
# Allow only one Foreman and one Puppet Master process
<IfModule mod_passenger.c>
PassengerMaxPoolSize 2
</IfModule>
EOPS

This limits number of processes one each, this will slow down processing of requests tho.

Comment 8 Bryan Kearney 2016-06-29 16:16:18 UTC

Moving to POST since upstream bug http://projects.theforeman.org/issues/15407 has been closed

Comment 10 Chris Duryee 2016-09-20 17:56:39 UTC

*** Bug 1334393 has been marked as a duplicate of this bug. ***

Comment 15 Roman Plevka 2016-12-05 14:15:56 UTC

VERIFIED
on satellite-6.2.5-1.0.el7sat.noarch

I used the docker containers (with katello-agent trigerring registration to the satellite). with ~200 hosts registering pretty much at the same time and was unable to hit the issue. (for the previous version of satellite, this was replicable even with <10 containers).

Comment 17 errata-xmlrpc 2016-12-12 17:11:46 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2940