Bug 1117376

Summary: Publish/Promotion times in Satellite 6 growing after each publish/promotion
Product: Red Hat Satellite Reporter: Alex Krzos <akrzos>
Component: Content ManagementAssignee: Justin Sherrill <jsherril>
Status: CLOSED ERRATA QA Contact: Tazim Kolhar <tkolhar>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.0.3CC: bbuckingham, cwelton, fvzwieten, jmontleo, jsherril, ldelouw, mhrivnak, mmccune, perfbz, tkolhar
Target Milestone: UnspecifiedKeywords: Performance, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
URL: http://projects.theforeman.org/issues/9647
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-08-12 05:10:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alex Krzos 2014-07-08 14:33:39 UTC
Description of problem:
Publishing and promoting content-views are growing in time required to publish/promote as more content-views are created.  Timing can be reduced by restarting foreman-proxy however not to the degree of a clean system.

Version-Release number of selected component (if applicable):
Satellite 6.0.3
RHEL 6.5 2.6.32-431.20.3
katello-1.5.0-26.el6sat.noarch
foreman-1.6.0.21-1.el6sat.noarch
candlepin-0.9.19-1.el6_5.noarch
pulp-server-2.4.0-0.23.beta.el6sat.noarch
puppet-3.6.2-1.el6sat.noarch


How reproducible:
Always

Steps to Reproduce:
1. Deploy Sat6 Server
2. Upload Manifest
3. Sync Content
4. Create, add repo, publish, promote more than 20 Content Views 
5. Review timings required to publish/promote each content-view

Actual results:
Here is 20 Content-View Publish Timings:
time hammer content-view publish --id $cvid
136.263
146.127
143.102
150.898
149.587
154.589
154.205
161.326
165.865
175.199
180.863
189.981
191.544
197.365
200.213
216.728
216.512
231.184
236.033
245.959


Expected results:
Content-view publishing should not grow as more

Additional info:
As a work around, restarting foreman-proxy reduces the timing but it continues to grow quickly there after.

Comment 1 RHEL Program Management 2014-07-08 14:44:22 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 3 Alex Krzos 2014-07-11 13:28:09 UTC
This also affects re-publishing:


The following shows the timing for a publish on the same content-view 10 times, then restarting foreman-proxy and followed by 10 more publishes on the same content-view.

128.119
126.899
142.159
153.075
167.625
179.12
188.724
197.218
216.234
228.535
# service foreman-proxy restart
130.982
144.965
143.197
156.973
167.467
180.09
187.695
202.582
214.01
223.046

Comment 4 Alex Krzos 2014-07-21 13:38:27 UTC
This also affects publishing of a content-view that only contains puppet-modules:

96.252
93.866
108.077
123.208
138.402
154.435
184.113
180.13
233.713
229.495
330.35
337.455
# service foreman-proxy restart
111.616
118.396
168.445
179.871
225.195
208.824
251.559
253.249
319.768
323.996
345.444
414.449
# service foreman-proxy restart
139.394
148.857
166.8
219.706
185.686
248.607

Additionally, the more and more content-views that are published/republished/promoted there appears to be no way to reduce the timing to its original state.  Restarting foreman-proxy helps somewhat however it doesn't restore it all the way down to the original.

Comment 5 Michael Hrivnak 2014-07-24 17:45:48 UTC
Working with Alex, we were able to determine that pulp is not responsible for the increasing times. All pulp tasks seem to be executing within consistent time frames.

Comment 6 Fred van Zwieten 2015-01-31 08:57:38 UTC
Although I have no concrete numbers, I see the same thing happening.

Comment 7 Justin Sherrill 2015-03-04 21:47:29 UTC
Created redmine issue http://projects.theforeman.org/issues/9647 from this bug

Comment 8 Justin Sherrill 2015-03-04 21:58:35 UTC
Opened a PR that hopefully addresses this issue.

I was not able to replicate exactly, but previously for every publish or promote we were fetching all puppet classes from all environments from the puppet master (we were only importing from the one environment we cared about).

The PR changes this to only fetch for the current relevant environment.  In my testing with ~12 puppet modules published in ~5 content views, this brought down the total publish time from 30 seconds down to 10 seconds.

I will leave it up to QA to validate that the timing does not increase quite as badly (since i was not able to reproduce exactly).

Comment 9 Bryan Kearney 2015-03-09 14:02:32 UTC
Moving to POST since upstream bug http://projects.theforeman.org/issues/9647 has been closed
-------------
Justin Sherrill
Applied in changeset commit:katello|a610d6c93a74e1e8c2ca2fdf930955ab99c07e99.

Comment 12 Tazim Kolhar 2015-03-13 10:26:23 UTC
VERIFIED :

# rpm -qa | grep foreman
rubygem-hammer_cli_foreman_tasks-0.0.3.3-1.el6_6sat.noarch
foreman-proxy-1.7.2.3-1.el6_6sat.noarch
foreman-ovirt-1.7.2.10-1.el6_6sat.noarch
ruby193-rubygem-foreman-tasks-0.6.12.1-1.el6_6sat.noarch
rubygem-hammer_cli_foreman_bootdisk-0.1.2.5-1.el6_6sat.noarch
ruby193-rubygem-foreman_abrt-0.0.5-2.el6_6sat.noarch
foreman-compute-1.7.2.10-1.el6_6sat.noarch
foreman-libvirt-1.7.2.10-1.el6_6sat.noarch
ruby193-rubygem-foreman_discovery-2.0.0.6-1.el6_6sat.noarch
rubygem-hammer_cli_foreman-0.1.4.6-1.el6_6sat.noarch
foreman-selinux-1.7.2.8-1.el6_6sat.noarch
foreman-gce-1.7.2.10-1.el6_6sat.noarch
ruby193-rubygem-foreman_docker-1.2.0.3-1.el6_6sat.noarch
ruby193-rubygem-foreman-redhat_access-0.0.9-1.el6_6sat.noarch
qe-sat6-rhel66.usersys.redhat.com-foreman-client-1.0-1.noarch
qe-sat6-rhel66.usersys.redhat.com-foreman-proxy-client-1.0-1.noarch
foreman-debug-1.7.2.10-1.el6_6sat.noarch
foreman-vmware-1.7.2.10-1.el6_6sat.noarch
ruby193-rubygem-foreman_bootdisk-4.0.2.8-1.el6_6sat.noarch
qe-sat6-rhel66.usersys.redhat.com-foreman-proxy-1.0-2.noarch
foreman-1.7.2.10-1.el6_6sat.noarch
ruby193-rubygem-foreman_hooks-0.3.7-2.el6_6sat.noarch
rubygem-hammer_cli_foreman_discovery-0.0.1.2-1.el6_6sat.noarch
ruby193-rubygem-foreman_gutterball-0.0.1.9-1.el6_6sat.noarch
foreman-postgresql-1.7.2.10-1.el6_6sat.noarch


# time hammer content-view publish --name con_view --organization 'Default Organization'
/usr/lib/ruby/gems/1.8/gems/hammer_cli-0.1.4/lib/hammer_cli/./apipie/../abstract.rb:68: warning: already initialized constant DEFAULT_LABEL_INDENT
[Foreman] Username: admin
[Foreman] Password for admin: 
[......................................................................] [100%]

real	0m13.538s
user	0m1.174s
sys	0m0.407s


# time hammer content-view publish --name con_view1 --organization 'Default Organization'
/usr/lib/ruby/gems/1.8/gems/hammer_cli-0.1.4/lib/hammer_cli/./apipie/../abstract.rb:68: warning: already initialized constant DEFAULT_LABEL_INDENT
[Foreman] Username: admin
[Foreman] Password for admin: 
[......................................................................] [100%]

real	0m12.241s
user	0m1.211s
sys	0m0.425s

Comment 13 Bryan Kearney 2015-03-31 19:44:20 UTC
*** Bug 1195979 has been marked as a duplicate of this bug. ***

Comment 14 Bryan Kearney 2015-08-11 13:29:31 UTC
This bug is slated to be released with Satellite 6.1.

Comment 17 errata-xmlrpc 2015-08-12 05:10:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:1592