Bug 1344708 - Duplicate key on "katello_content_facet_errata_eid_caid" when Satellite 6.2 was put under load
Summary: Duplicate key on "katello_content_facet_errata_eid_caid" when Satellite 6.2 w...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: WebUI
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified vote
Target Milestone: Unspecified
Assignee: Justin Sherrill
QA Contact: Roman Plevka
URL: http://projects.theforeman.org/issues...
Whiteboard:
: 1334393 (view as bug list)
Depends On:
Blocks: 1383481
TreeView+ depends on / blocked
 
Reported: 2016-06-10 12:38 UTC by Lukas Zapletal
Modified: 2019-08-12 14:08 UTC (History)
10 users (show)

Fixed In Version: tfm-rubygem-katello-3.0.0.86-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1383481 (view as bug list)
Environment:
Last Closed: 2016-12-12 17:11:46 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Foreman Issue Tracker 15407 Normal Closed Duplicate key on "katello_content_facet_errata_eid_caid" under load 2020-07-08 19:44:31 UTC
Red Hat Product Errata RHBA-2016:2940 normal SHIPPED_LIVE Satellite 6.2.5 Async Bug Release 2016-12-12 22:08:21 UTC

Description Lukas Zapletal 2016-06-10 12:38:51 UTC
Reproducer:

Create several processes that register content hosts (subscription-manager register with an activation key with some content associated) in parallel.

2016-06-10 10:50:29 CEST ERROR:  duplicate key value violates unique constraint                      +"katello_content_facet_errata_eid_caid"
2016-06-10 10:50:29 CEST DETAIL:  Key (erratum_id, content_facet_id)=(18, 122) already exists.
2016-06-10 10:50:29 CEST STATEMENT:  INSERT INTO katello_content_facet_errata (erratum_id,           +content_facet_id) VALUES (18, 122),....
2016-06-10 10:50:29 CEST ERROR:  current transaction is aborted, commands ignored until end of       +transaction block
2016-06-10 10:50:29 CEST STATEMENT:  DELETE FROM "katello_content_facet_errata" WHERE                +"katello_content_facet_errata"."content_facet_id" = 122

2016-06-10 10:56:40 CEST ERROR:  duplicate key value violates unique constraint                      +"katello_content_facet_errata_eid_caid"
2016-06-10 10:56:40 CEST DETAIL:  Key (erratum_id, content_facet_id)=(9, 131) already exists.
2016-06-10 10:56:40 CEST STATEMENT:  INSERT INTO katello_content_facet_errata (erratum_id,           +content_facet_id) VALUES ...
2016-06-10 10:56:40 CEST ERROR:  current transaction is aborted, commands ignored until end of       +transaction block
2016-06-10 10:56:40 CEST STATEMENT:  DELETE FROM "katello_content_facet_errata" WHERE                +"katello_content_facet_errata"."content_facet_id" = 131

2016-06-10 11:05:13 CEST ERROR:  duplicate key value violates unique constraint                      +"katello_content_facet_errata_eid_caid"
2016-06-10 11:05:13 CEST DETAIL:  Key (erratum_id, content_facet_id)=(10, 149) already exists.
2016-06-10 11:05:13 CEST STATEMENT:  INSERT INTO katello_content_facet_errata (erratum_id,           +content_facet_id) VALUES ...
2016-06-10 11:05:13 CEST ERROR:  current transaction is aborted, commands ignored until end of       +transaction block
2016-06-10 11:05:13 CEST STATEMENT:  DELETE FROM "katello_content_facet_errata" WHERE                +"katello_content_facet_errata"."content_facet_id" = 149

2016-06-10 11:09:21 CEST ERROR:  duplicate key value violates unique constraint                      +"katello_content_facet_errata_eid_caid"
2016-06-10 11:09:21 CEST DETAIL:  Key (erratum_id, content_facet_id)=(1566, 147) already exists.
2016-06-10 11:09:21 CEST STATEMENT:  INSERT INTO katello_content_facet_errata (erratum_id,           +content_facet_id) VALUES (1566, 147), (2298, 147), (424, 147), (2828, 147), (2903, 147), (637,      +147), (3931, 147), (4011, 147), (4340, 147)
2016-06-10 11:09:21 CEST ERROR:  current transaction is aborted, commands ignored until end of       +transaction block
2016-06-10 11:09:21 CEST STATEMENT:  DELETE FROM "katello_content_facet_errata" WHERE                +"katello_content_facet_errata"."content_facet_id" = 147

# rpm -q satellite katello foreman
satellite-6.2.0-14.2.el7sat.noarch
katello-3.0.0-6.el7sat.noarch
foreman-1.11.0.34-1.el7sat.noarch

Debug tarball (Foreman core team permissions needed to download it):
http://debugs.theforeman.org/foreman-debug-0Qsen.tar.xz

I am able to randomly reproduce on my own box, takes hours to hit with about 100 clients registering.

Comment 1 Lukas Zapletal 2016-06-10 12:44:43 UTC
Note: You may need to configure PassengerMinInstances 2 in /etc/httpd/conf.d/05-foreman-ssl.conf in order to have at least two processes that interfere each other.

Comment 5 Justin Sherrill 2016-06-14 12:45:54 UTC
Created redmine issue http://projects.theforeman.org/issues/15407 from this bug

Comment 6 Bryan Kearney 2016-06-14 14:17:22 UTC
Upstream bug component is WebUI

Comment 7 Lukas Zapletal 2016-06-20 12:32:54 UTC
WORKAROUND:

cat >>/etc/httpd/conf.d/passenger.conf <<EOPS
# Allow only one Foreman and one Puppet Master process
<IfModule mod_passenger.c>
PassengerMaxPoolSize 2
</IfModule>
EOPS

This limits number of processes one each, this will slow down processing of requests tho.

Comment 8 Bryan Kearney 2016-06-29 16:16:18 UTC
Moving to POST since upstream bug http://projects.theforeman.org/issues/15407 has been closed

Comment 10 Chris Duryee 2016-09-20 17:56:39 UTC
*** Bug 1334393 has been marked as a duplicate of this bug. ***

Comment 15 Roman Plevka 2016-12-05 14:15:56 UTC
VERIFIED
on satellite-6.2.5-1.0.el7sat.noarch

I used the docker containers (with katello-agent trigerring registration to the satellite). with ~200 hosts registering pretty much at the same time and was unable to hit the issue. (for the previous version of satellite, this was replicable even with <10 containers).

Comment 17 errata-xmlrpc 2016-12-12 17:11:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2940


Note You need to log in before you can comment on or make changes to this bug.