Bug 1758285

Summary: [Tracker][scale & performance] Move Actions::Katello::Host::* tasks out of Dynflow & Foreman Tasks
Product: Red Hat Satellite Reporter: Mike McCune <mmccune>
Component: Hosts - ContentAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED CURRENTRELEASE QA Contact: Stephen Wadeley <swadeley>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.6.0CC: awyatt, dmule, inecas, ktordeur, momran, paji, patalber, sshtein, wpinheir
Target Milestone: UnspecifiedKeywords: Performance, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-07-08 08:33:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1771719, 1771724, 1771729    
Bug Blocks:    

Description Mike McCune 2019-10-03 18:33:20 UTC
Due to the extraordinarily large volume of calls from subscription-manager creating the following 3 task types:

Actions::Katello::Host::Update
Actions::Katello::Host::GenerateApplicability
Actions::Katello::Host::UploadPackageProfile

We often see a 100:1 ratio of this type of task vs other task types in the database, an example of a user who had 700,000 tasks in their task history saw this rough breakdown

  COUNT|TYPE 

 320668 Actions::Katello::Host::Update
 277923 Actions::Katello::Host::GenerateApplicability
 141833 Actions::Katello::Host::UploadPackageProfile
   1983 Actions::Katello::Repository::ScheduledSync
    429 Actions::Katello::CapsuleContent::Sync
    127 Actions::Katello::Repository::MetadataGenerate
    118 Actions::Katello::Repository::CapsuleGenerateAndSync
    109 Actions::Katello::CapsuleContent::RemoveOrphans
     59 Actions::Katello::Host::Hypervisors
     53 Actions::Katello::Repository::BulkMetadataGenerate
...

we need to investigate moving these to be handled directly by the main Rails application and not require the overhead of task creation, monitoring and eventual database cleanup.

We saw performance improvements when we moved 'subscription-manager register' calls to be handled directly in Foreman and want to consider the same for the above task types.

Also take into consideration we need to allow for concurrent calls as added in this bz here in 6.5.z and 6.6:

https://bugzilla.redhat.com/show_bug.cgi?id=1643649

Comment 3 Partha Aji 2019-11-12 20:29:52 UTC
I decided to use this as a tracker and create Bugs for
1) https://bugzilla.redhat.com/show_bug.cgi?id=1771719 - Move Actions::Katello::Host::Update out of dynflow
2) https://bugzilla.redhat.com/show_bug.cgi?id=1771729 - Move Actions::Katello::Host::GenerateApplicability out of dynflow
3) https://bugzilla.redhat.com/show_bug.cgi?id=1771724 - Move Actions::Katello::Host::UploadPackageProfile out of dynflow

Comment 4 Mike McCune 2019-11-13 18:06:59 UTC
Analysis of customer sosreports shows our largest environments often show Actions::Katello::Host::Update calls ranging in the 30,000 to 50,000 tasks per day.

Comment 5 Stephen Wadeley 2022-07-08 08:33:02 UTC
Hello

as all the bugs in comment 3 are resolved I will close this as CURRENT RELEASE.