Bug 1684653

Summary: Content view publish planning in foreman-tasks sometimes takes too long
Product: Red Hat Satellite Reporter: sthirugn <sthirugn>
Component: Content ViewsAssignee: Justin Sherrill <jsherril>
Status: CLOSED NOTABUG QA Contact: Lai <ltran>
Severity: medium Docs Contact:
Priority: high    
Version: 6.4.2CC: bbuckingham, jsherril, wclark
Target Milestone: UnspecifiedKeywords: Performance, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-31 21:23:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
content view publish planning none

Description sthirugn@redhat.com 2019-03-01 18:50:09 UTC 
Created attachment 1539890 [details]
content view publish planning

Description of problem:
Content view publish planning in foreman-tasks sometimes takes too long

Version-Release number of selected component (if applicable):
Satellite-6.3.5
Satellite-6.4.2

How reproducible:
Always

Steps to Reproduce:
1.Create a content view with around 100 yum repos
2.Publish the content view

Actual results:
Content View publish task planning takes a long time.  For a similar content view, time taken for publish task planning (see screenshot on where to find the planning time for publish task) are:
Satellite 6.3.5 - 75 seconds
Satellite 6.4.2 - 75 seconds
Note that this test happened with zero client traffic.  Having client traffic to the satellite further increases the planning time.

Because of this hammer cv publish users have to wait for a long time to get the foreman task id back. In the example below, it took almost 2 minutes to get the task id back on a satellite server with no client traffic.

# date; hammer content-view publish --name rhel7cv --organization-id=21 --async; date
Fri Mar  1 13:43:13 EST 2019
Content view is being published with task e1fff9b2-3060-479a-be45-71d3ab54d9e4
Fri Mar  1 13:45:06 EST 2019

Expected results:
CV publish planning should happen faster and the task id for the hammer command should be returned sooner.

Additional info:
A similar bug for CV capsule sync task planning was fixed in BZ1673447
Comment 8 Justin Sherrill 2019-07-26 18:29:41 UTC 
The attached customer case of 15 repos taking ~5 minutes is wildly different than publishing 100 repos taking ~85 seconds.  Looking at the sosreport, it looks like their dynflow_tables have quite a bit of data in them. 

Can we try clearing out some of the old tasks and try the publish again?  If that doesn't improve things dramatically, lets grab a database backup and see if we can reproduce their times in house.

Satellite 6.6 should have some improvements due to task restructuring within CV Publish, and i believe there is a small change we can make to improve the ~100 repo scenario (but likely won't help a ton with ~15 repos).  I wouldn't expect a huge difference from these changes though.
Comment 10 Justin Sherrill 2020-07-31 21:23:22 UTC 
I'm going to close this out as the customer case is closed and we have not been able to reproduce.  Please reopen if there is still an issue!