Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1344708 - Duplicate key on "katello_content_facet_errata_eid_caid" when Satellite 6.2 was put under load
Duplicate key on "katello_content_facet_errata_eid_caid" when Satellite 6.2 w...
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: WebUI (Show other bugs)
6.2.0
Unspecified Unspecified
unspecified Severity unspecified (vote)
: 6.2.5
: Unused
Assigned To: Justin Sherrill
Roman Plevka
http://projects.theforeman.org/issues...
: Triaged
: 1334393 (view as bug list)
Depends On:
Blocks: 1383481
  Show dependency treegraph
 
Reported: 2016-06-10 08:38 EDT by Lukas Zapletal
Modified: 2016-12-12 12:11 EST (History)
10 users (show)

See Also:
Fixed In Version: tfm-rubygem-katello-3.0.0.86-1
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1383481 (view as bug list)
Environment:
Last Closed: 2016-12-12 12:11:46 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Foreman Issue Tracker 15407 None None None 2016-06-14 09:06 EDT
Red Hat Product Errata RHBA-2016:2940 normal SHIPPED_LIVE Satellite 6.2.5 Async Bug Release 2016-12-12 17:08:21 EST

  None (edit)
Description Lukas Zapletal 2016-06-10 08:38:51 EDT
Reproducer:

Create several processes that register content hosts (subscription-manager register with an activation key with some content associated) in parallel.

2016-06-10 10:50:29 CEST ERROR:  duplicate key value violates unique constraint                      +"katello_content_facet_errata_eid_caid"
2016-06-10 10:50:29 CEST DETAIL:  Key (erratum_id, content_facet_id)=(18, 122) already exists.
2016-06-10 10:50:29 CEST STATEMENT:  INSERT INTO katello_content_facet_errata (erratum_id,           +content_facet_id) VALUES (18, 122),....
2016-06-10 10:50:29 CEST ERROR:  current transaction is aborted, commands ignored until end of       +transaction block
2016-06-10 10:50:29 CEST STATEMENT:  DELETE FROM "katello_content_facet_errata" WHERE                +"katello_content_facet_errata"."content_facet_id" = 122

2016-06-10 10:56:40 CEST ERROR:  duplicate key value violates unique constraint                      +"katello_content_facet_errata_eid_caid"
2016-06-10 10:56:40 CEST DETAIL:  Key (erratum_id, content_facet_id)=(9, 131) already exists.
2016-06-10 10:56:40 CEST STATEMENT:  INSERT INTO katello_content_facet_errata (erratum_id,           +content_facet_id) VALUES ...
2016-06-10 10:56:40 CEST ERROR:  current transaction is aborted, commands ignored until end of       +transaction block
2016-06-10 10:56:40 CEST STATEMENT:  DELETE FROM "katello_content_facet_errata" WHERE                +"katello_content_facet_errata"."content_facet_id" = 131

2016-06-10 11:05:13 CEST ERROR:  duplicate key value violates unique constraint                      +"katello_content_facet_errata_eid_caid"
2016-06-10 11:05:13 CEST DETAIL:  Key (erratum_id, content_facet_id)=(10, 149) already exists.
2016-06-10 11:05:13 CEST STATEMENT:  INSERT INTO katello_content_facet_errata (erratum_id,           +content_facet_id) VALUES ...
2016-06-10 11:05:13 CEST ERROR:  current transaction is aborted, commands ignored until end of       +transaction block
2016-06-10 11:05:13 CEST STATEMENT:  DELETE FROM "katello_content_facet_errata" WHERE                +"katello_content_facet_errata"."content_facet_id" = 149

2016-06-10 11:09:21 CEST ERROR:  duplicate key value violates unique constraint                      +"katello_content_facet_errata_eid_caid"
2016-06-10 11:09:21 CEST DETAIL:  Key (erratum_id, content_facet_id)=(1566, 147) already exists.
2016-06-10 11:09:21 CEST STATEMENT:  INSERT INTO katello_content_facet_errata (erratum_id,           +content_facet_id) VALUES (1566, 147), (2298, 147), (424, 147), (2828, 147), (2903, 147), (637,      +147), (3931, 147), (4011, 147), (4340, 147)
2016-06-10 11:09:21 CEST ERROR:  current transaction is aborted, commands ignored until end of       +transaction block
2016-06-10 11:09:21 CEST STATEMENT:  DELETE FROM "katello_content_facet_errata" WHERE                +"katello_content_facet_errata"."content_facet_id" = 147

# rpm -q satellite katello foreman
satellite-6.2.0-14.2.el7sat.noarch
katello-3.0.0-6.el7sat.noarch
foreman-1.11.0.34-1.el7sat.noarch

Debug tarball (Foreman core team permissions needed to download it):
http://debugs.theforeman.org/foreman-debug-0Qsen.tar.xz

I am able to randomly reproduce on my own box, takes hours to hit with about 100 clients registering.
Comment 1 Lukas Zapletal 2016-06-10 08:44:43 EDT
Note: You may need to configure PassengerMinInstances 2 in /etc/httpd/conf.d/05-foreman-ssl.conf in order to have at least two processes that interfere each other.
Comment 5 Justin Sherrill 2016-06-14 08:45:54 EDT
Created redmine issue http://projects.theforeman.org/issues/15407 from this bug
Comment 6 Bryan Kearney 2016-06-14 10:17:22 EDT
Upstream bug component is WebUI
Comment 7 Lukas Zapletal 2016-06-20 08:32:54 EDT
WORKAROUND:

cat >>/etc/httpd/conf.d/passenger.conf <<EOPS
# Allow only one Foreman and one Puppet Master process
<IfModule mod_passenger.c>
PassengerMaxPoolSize 2
</IfModule>
EOPS

This limits number of processes one each, this will slow down processing of requests tho.
Comment 8 Bryan Kearney 2016-06-29 12:16:18 EDT
Moving to POST since upstream bug http://projects.theforeman.org/issues/15407 has been closed
Comment 10 Chris Duryee 2016-09-20 13:56:39 EDT
*** Bug 1334393 has been marked as a duplicate of this bug. ***
Comment 15 Roman Plevka 2016-12-05 09:15:56 EST
VERIFIED
on satellite-6.2.5-1.0.el7sat.noarch

I used the docker containers (with katello-agent trigerring registration to the satellite). with ~200 hosts registering pretty much at the same time and was unable to hit the issue. (for the previous version of satellite, this was replicable even with <10 containers).
Comment 17 errata-xmlrpc 2016-12-12 12:11:46 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2940

Note You need to log in before you can comment on or make changes to this bug.