Bug 1679241

Summary: Bulk attaching subscriptions triggers foreman database deadlocks
Product: Red Hat Satellite Reporter: Julio Entrena Perez <jentrena>
Component: Subscription ManagementAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED INSUFFICIENT_DATA QA Contact: jcallaha
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.4.1CC: cmarinea, jentrena
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-17 09:30:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1115190    

Description Julio Entrena Perez 2019-02-20 16:37:39 UTC
Description of problem:
Bulk attaching subscriptions to a large number of hosts results in foreman database deadlocks on table katello_pools for some of the dynflow tasks.
For example, bulk attaching subscriptions to 150 hosts results in 125 successful tasks and 25 failed tasks with database deadlocks:

2019-02-15 11:07:55 GMT ERROR:  deadlock detected
2019-02-15 11:07:55 GMT DETAIL:  Process 438816 waits for ExclusiveLock on tuple (1,19) of relation 19806 of database 18080; blocked by process 439677.
        Process 439677 waits for ShareLock on transaction 94135696; blocked by process 438814.
        Process 438814 waits for ShareLock on transaction 94136055; blocked by process 438816.
        Process 438816: UPDATE "katello_pools" SET "updated_at" = $1, "consumed" = $2 WHERE "katello_pools"."id" = $3
        Process 439677: UPDATE "katello_pools" SET "updated_at" = $1, "consumed" = $2 WHERE "katello_pools"."id" = $3
        Process 438814: UPDATE "katello_pools" SET "updated_at" = $1, "consumed" = $2 WHERE "katello_pools"."id" = $3
2019-02-15 11:07:55 GMT HINT:  See server log for query details.
2019-02-15 11:07:55 GMT STATEMENT:  UPDATE "katello_pools" SET "updated_at" = $1, "consumed" = $2 WHERE "katello_pools"."id" = $3

Version-Release number of selected component (if applicable):
satellite-6.4.1-1.el7sat

How reproducible:
Always

Steps to Reproduce:
1. Hosts -> Content Hosts -> select 150 hosts -> Select Action -> Manage Subscriptions
2. In Content Host Bulk Subscriptions window, select 4 subscriptions and click Done button.
3. Inspect the resulting tasks and the PostgreSQL logs.

Actual results:
125 hosts have the 4 subscriptions attached to them.
25 hosts fail, 25 "Actions::Katello::Host::AttachSubscriptions" tasks end with warnings, details of the Finalize tab of the dynflow task show database deadlock.

Expected results:
All 150 tasks succeed, all 150 hosts have 4 subscriptions attached, no database deadlocks.

Additional info: