Bug 1293911

Summary: Concurrent Actions::Katello::System::GenerateApplicability tasks fail in paused with DB transacation abort
Product: Red Hat Satellite Reporter: Peter Vreman <peter.vreman>
Component: Errata ManagementAssignee: Justin Sherrill <jsherril>
Status: CLOSED ERRATA QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: 6.1.5CC: bbuckingham, bkearney, cduryee, chrobert, ebeaudoi, erik-fedora, jjansky, jsherril, lzap, mlinden, mmccune, mtaru, oshtaier, pmoravec, rplevka, sauchter, sthirugn
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-27 11:14:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1122832, 1317008    

Description Peter Vreman 2015-12-23 13:33:03 UTC
Description of problem:
The task GenerateApplicability for a system cannot handle concurrent requests. E.g. running multiple separate yum commands on a client after eachother multiple requests to upload the package profile will be done. 

This leads then to paused tasks becuase of failing database transactions:

PGError: ERROR:  current transaction is aborted, commands ignored until end of transaction block
: DELETE FROM "katello_system_errata" WHERE "katello_system_errata"."system_id" = 155

Id: e254b260-24c2-42ba-a119-75152a315745
Label: Actions::Katello::System::GenerateApplicability
Name: Generate applicability
Owner: foreman_admin
Started at: 2015-12-21 20:09:05 UTC
Ended at: 2015-12-21 20:12:51 UTC
State: stopped
Result: success
Params: {"services_checked"=>["pulp", "pulp_auth"], "system_ids"=>[155], "locale"=>"en"}



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. On a client execute yum command quickly after each other so it uplaods the package profile.
Example below:
for i in `seq 1 5`; do yum install -y bash-doc ; yum remove -y bash-doc; done
2.
3.

Actual results:
paused tasks

Expected results:
No paused tasks


Additional info:

Comment 6 Pavel Moravec 2016-05-17 11:01:50 UTC
Also hit in 6.1.8. pgsql denies the transaction due to pgsql error in logs:

ERROR:  duplicate key value violates unique constraint "katello_system_errata_eid_sid"
DETAIL:  Key (erratum_id, system_id)=(10222, 710) already exists.
STATEMENT:  INSERT INTO katello_system_errata (erratum_id, system_id) VALUES (10222, 710), (10223, 710), (10224, 710), (10287, 710), (10288, 710), (10290, 710), (10291, 710), (10293, 710), (10294, 710), (10296, 710), (10300, 710), (10302, 710), (10303, 710), (10304, 710), (10305, 710), (10306, 710), (10307, 710), (10308, 710), (10202, 710), (10347, 710), (10206, 710), (10153, 710), (10154, 710), (10162, 710), (10164, 710), (10165, 710), (10166, 710), (10167, 710), (10168, 710), (10170, 710), (10176, 710), (10177, 710), (10180, 710), (10181, 710), (10182, 710), (10185, 710), (10186, 710), (10327, 710), (10415, 710), (10256, 710), (10258, 710), (10344, 710)

Does it raise a bell? Why katello tries to insert something that is (partially) in the db already?

Comment 11 Bryan Kearney 2016-05-31 13:09:58 UTC
MOving to 6.2, verified. Dev believes 6.2 should address this and QE was not able to re-produce on the 6.2 beta code.

Comment 12 Lukas Zapletal 2016-06-07 11:00:41 UTC
I reproduced this once again today with 6.2 beta2 drop:

[root@ibm-ls22-03 ~]# rpm -q satellite katello foreman
satellite-6.2.0-14.2.el7sat.noarch
katello-3.0.0-6.el7sat.noarch
foreman-1.11.0.34-1.el7sat.noarch

I have a system where I encountered the error once during client testing.

Actions::Katello::Host::GenerateApplicability

PG::Error: ERROR:  current transaction is aborted, commands ignored until end of transaction block: DELETE FROM "katello_content_facet_errata" WHERE "katello_content_facet_errata"."content_facet_id" = 22

Resuming the task solved the issue.

Comment 14 Lukas Zapletal 2016-06-07 11:11:57 UTC
For the record, SQL statements which caused this are:

2016-06-07 09:23:51 CEST ERROR:  duplicate key value violates unique constraint "katello_content_facet_errata_eid_caid"
2016-06-07 09:23:51 CEST DETAIL:  Key (erratum_id, content_facet_id)=(18, 15) already exists.
2016-06-07 09:23:51 CEST STATEMENT:  INSERT INTO katello_content_facet_errata (erratum_id, content_facet_id) VALUES (18, 15), (107, 15), (158, 15), (210, 15), (253, 15), (261, 15), (321, 15), (439, 15), (404, 15), (424, 15), (483, 15), (561, 15), (589, 15), (613, 15), (699, 15), (774, 15), (833, 15), (799, 15), (849, 15), (867, 15), (914, 15), (916, 15), (964, 15), (989, 15), (1154, 15), (5643, 15), (1690, 15), (5671, 15), (5343, 15), (6242, 15), (5573, 15), (5451, 15), (5811, 15), (5834, 15), (4916, 15), (4945, 15), (5011, 15), (5874, 15), (5051, 15), (5213, 15), (5248, 15), (5256, 15), (5264, 15), (5272, 15), (6160, 15), (5479, 15), (5526, 15), (6030, 15), (6066, 15), (6092, 15), (6178, 15), (6180, 15), (6261, 15), (6350, 15), (6360, 15), (6429, 15), (6548, 15), (6569, 15), (6639, 15), (6663, 15), (6760, 15), (6790, 15), (6808, 15), (7727, 15), (6963, 15), (7177, 15), (7192, 15), (7203, 15), (7296, 15), (7320, 15), (7331, 15), (7368, 15), (7377, 15), (7410, 15), (7421, 15), (7432, 15), (7463, 15), (7479, 15), (7545, 15), (7618, 15), (7635, 15)
2016-06-07 09:23:51 CEST ERROR:  current transaction is aborted, commands ignored until end of transaction block
2016-06-07 09:23:51 CEST STATEMENT:  DELETE FROM "katello_content_facet_errata" WHERE "katello_content_facet_errata"."content_facet_id" = 15

2016-06-07 09:40:31 CEST ERROR:  duplicate key value violates unique constraint "katello_content_facet_errata_eid_caid"
2016-06-07 09:40:31 CEST DETAIL:  Key (erratum_id, content_facet_id)=(1566, 22) already exists.
2016-06-07 09:40:31 CEST STATEMENT:  INSERT INTO katello_content_facet_errata (erratum_id, content_facet_id) VALUES (1566, 22), (2298, 22), (424, 22), (2828, 22), (2903, 22), (637, 22), (3931, 22), (4011, 22), (4340, 22)
2016-06-07 09:40:31 CEST ERROR:  current transaction is aborted, commands ignored until end of transaction block
2016-06-07 09:40:31 CEST STATEMENT:  DELETE FROM "katello_content_facet_errata" WHERE "katello_content_facet_errata"."content_facet_id" = 2

Makes me to believe itt's the constraint "katello_content_facet_errata_eid_caid" which makes this to fail.

Comment 18 Justin Sherrill 2016-06-13 13:03:49 UTC
Moving this back to verified as https://bugzilla.redhat.com/show_bug.cgi?id=1344708 was opened to address comment #11 and QE no longer sees this issue (and its slightly different from comment #11).

Comment 19 Bryan Kearney 2016-07-27 11:14:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501