Bug 1522912 (cv-shredder)

Summary: Add content view publish and promotion optimizations
Product: Red Hat Satellite Reporter: Justin Sherrill <jsherril>
Component: Content ViewsAssignee: Justin Sherrill <jsherril>
Status: CLOSED ERRATA QA Contact: jcallaha
Severity: high Docs Contact:
Priority: high    
Version: 6.3.0CC: amdas, andrew.schofield, bbuckingham, bkearney, brubisch, cdonnell, ehelms, fgarciad, kdixon, mmccune, peter.vreman, riehecky, rjerrido, rsevilla, stbenjam
Target Milestone: UnspecifiedKeywords: PrioBumpGSS, PrioBumpPM, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tfm-rubygem-katello-3.4.5.60-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-13 13:29:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1122832, 1190823    
Attachments:
Description Flags
6.3 Beta hotfix packages for Content View performance improvement
none
6.3.0 Hotfix none

Description Justin Sherrill 2017-12-06 17:43:56 UTC 
Description of problem:

Upstream katello has introduced a number of changes to increase the performance of content view promotion and publishing. 

it can decrease promotion times by ~99%, normal publishing times by 30-50%, and composite publish times by 99%.

These should be backported to 6.3 if desired.
Comment 1 Justin Sherrill 2017-12-06 17:58:32 UTC 
The work is being tracked upstream here: http://projects.theforeman.org/issues/21698
Comment 3 Satellite Program 2017-12-11 21:06:01 UTC 
Upstream bug assigned to jsherril
Comment 4 Satellite Program 2017-12-11 21:06:05 UTC 
Upstream bug assigned to jsherril
Comment 5 Mike McCune 2017-12-21 15:44:15 UTC 
Created attachment 1370954 [details]
6.3 Beta hotfix packages for Content View performance improvement
Comment 6 Mike McCune 2017-12-21 15:44:49 UTC 
*** Satellite 6.3 Beta Update Packages *** 

We are including packages in this bug that includes a performance enhancement to Content View operations in 6.3 Beta. This performance enhancement greatly reduces the publish and promotion times for Content Views. Some operations will see multiple orders of magnitude of time reduction to complete.

These package update also include a performance fix to Bug:

* Bug 1509965 - very slow publishing of a content view with filters containing many errata 

Some observations from our testing of this improvement:

* In test environments we observed publishing of a Content View time reduced by 30-50% depending on hardware and environment.

* In test environments we have observed publish times for Composite Content Views reduced by over 99%, often complete in under a minute

* In test environments promotion times for Content Views (Composite and standard) reduced by over 99%, often complete in under a minute

* The first publish and promotion of existing Content Views from prior to patching may not see as big of a speed increase but subsequent publish/promotions will see maximum benefit.

We encourage anyone with Satellite 6.3 Beta to download and install these packages on their 6.3 Beta installation and provide feedback via comments in this bug or our beta list:

satellite6-beta <satellite6-beta>

INSTALLATION INSTRUCTIONS:

1) Download attached content-views-perf-63-beta-1522912.tar containing hotfix packages

2) Unpack 

3) katello-service stop

4) rpm -Uvh tfm-rubygem-katello*.rpm

5) katello-service start

Resume operations

NOTE: These packages will only work with Satellite 6.3 Beta and no other version of Satellite 6. This performance enhancement will ship in a future update to Satellite 6.3.Z in 2018.
Comment 7 Amit Kumar Das 2017-12-22 07:03:06 UTC 
My observation on sat6.3-beta hosted on RHEL7.4(2cpu,16gb RAM). After applying performance enhancement package, 30-40% time reduced for publishing CV(repo-RHEL7Server 17802 Packages) and 99% for promoting the CV.

Label: Actions::Katello::ContentView::Publish
Started at: 2017-12-22 05:18:53 UTC         <-approx 6 mins
Ended at:   2017-12-22 05:24:46 UTC 
Label: Actions::Katello::ContentView::Promote
Started at: 2017-12-22 05:27:12 UTC         <--approx 1.5 mins
Ended at:   2017-12-22 05:28:57 UTC 

//AFTER APPLYING PATCH//
Label: Actions::Katello::ContentView::Publish
Started at: 2017-12-22 06:01:50 UTC         <-approx 4 mins
Ended at:   2017-12-22 06:06:07 UTC 
Label: Actions::Katello::ContentView::Promote
Started at: 2017-12-22 06:07:44 UTC         <-2 sec
Ended at:   2017-12-22 06:07:49 UTC 


==CV WITH ERRATA FILTER BY DATE==
Label: Actions::Katello::ContentView::Publish
Started at: 2017-12-22 05:35:24 UTC         <-approx 9 mins
Ended at:   2017-12-22 05:44:07 UTC 
Label: Actions::Katello::ContentView::Promote
Started at: 2017-12-22 05:50:54 UTC         <-approx 1.5 mins
Ended at:   2017-12-22 05:52:22 UTC 

//AFTER APPLYING PATCH//
Label: Actions::Katello::ContentView::Publish
Started at: 2017-12-22 06:17:59 UTC         <-approx 6 mins
Ended at:   2017-12-22 06:25:25 UTC 
Label: Actions::Katello::ContentView::Promote
Started at: 2017-12-22 06:30:45 UTC         <-7 sec 
Ended at:   2017-12-22 06:30:51 UTC
Comment 8 Justin Sherrill 2018-02-19 16:18:12 UTC 
Created attachment 1397938 [details]
6.3.0 Hotfix
Comment 9 Justin Sherrill 2018-02-19 16:38:59 UTC 
*** Satellite 6.3 Packages *** 

We are including packages in this bug that includes a performance enhancement to Content View operations in 6.3. This performance enhancement greatly reduces the publish and promotion times for Content Views. Some operations will see multiple orders of magnitude of time reduction to complete.

These package update also include a performance fix to Bug:

* Bug 1509965 - very slow publishing of a content view with filters containing many errata 

Some observations from our testing of this improvement:

* In test environments we observed publishing of a Content View time reduced by 30-50% depending on hardware and environment.

* In test environments we have observed publish times for Composite Content Views reduced by over 99%, often complete in under a minute

* In test environments promotion times for Content Views (Composite and standard) reduced by over 99%, often complete in under a minute

* The first publish and promotion of existing Content Views from prior to patching may not see as big of a speed increase but subsequent publish/promotions will see maximum benefit.


INSTALLATION INSTRUCTIONS:

1) Download attached 6.3.0-hotfix.tar containing hotfix packages

2) Unpack 

3) katello-service stop

4) rpm -Uvh tfm-rubygem-katello*.rpm

5) katello-service start

Resume operations

NOTE: These packages will only work with Satellite 6.3.0 and no other version of Satellite 6. This performance enhancement will ship in a future update to Satellite 6.3.Z in 2018.
Comment 10 Justin Sherrill 2018-03-05 15:30:53 UTC 
*** Bug 1184853 has been marked as a duplicate of this bug. ***
Comment 11 Peter Vreman 2018-03-05 16:02:45 UTC 
All my CVs and CCVs have puppet modules. When i tried to publish a new version of a CCVs i got:

2018-03-05 15:58:57 b7d2bccf [foreman-tasks/action] [E] Validation failed: Name has already been taken (ActiveRecord::RecordInvalid)

from 

 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.58/app/lib/actions/katello/content_view_puppet_environment/clone.rb:48:in `setup_puppet_environment_clone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.58/app/lib/actions/katello/content_view_puppet_environment/clone.rb:25:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan.rb:350:in `switch_flow'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:381:in `sequence'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.58/app/lib/actions/katello/content_view_puppet_environment/clone.rb:21:in `plan'
Comment 12 Peter Vreman 2018-03-05 18:21:10 UTC 
Forget the remark in comment 11, that is a common 6.3.0 problem that i am unable to publish through the UI, but through the CLI/API it works.
Comment 13 Peter Vreman 2018-03-07 15:37:59 UTC 
A happy consumer result:

-----
[Unknown Hilti Tier] root@li-lc-1578:~# sudo -u hoici hammer -c /opt/hoici/etc/sat6/hammer-hoici.yaml task list --search 'state == stopped and label == Actions::Katello::ContentView::Publish'
-------------------------------------|------|-------|---------------------|---------------------|---------|---------|-------------|------------
ID                                   | NAME | OWNER | STARTED AT          | ENDED AT            | STATE   | RESULT  | TASK ACTION | TASK ERRORS
-------------------------------------|------|-------|---------------------|---------------------|---------|---------|-------------|------------
7616b738-4813-48db-b9f6-60384aa929bc |      | hoici | 2018/03/07 15:09:05 | 2018/03/07 15:09:32 | stopped | success | Publish     |
bb91c8cb-0d1b-4c19-908c-d42d1101e202 |      | hoici | 2018/03/07 15:09:03 | 2018/03/07 15:09:31 | stopped | success | Publish     |
6c9dc4c7-a2ed-4fd4-bb0e-cb9fa1c651e7 |      | hoici | 2018/03/07 14:31:34 | 2018/03/07 14:31:59 | stopped | success | Publish     |
43b0b865-828d-4218-828d-2f9c6d59bcf4 |      | hoici | 2018/03/07 14:30:34 | 2018/03/07 14:30:56 | stopped | success | Publish     |
38bae128-1460-43b9-9f40-3c233ae67d4f |      | hoici | 2018/03/07 14:29:34 | 2018/03/07 14:29:57 | stopped | success | Publish     |
03bc86e2-72c0-4015-8327-39b09aa9d6ba |      | hoici | 2018/03/07 14:28:34 | 2018/03/07 14:28:57 | stopped | success | Publish     |
0c31c8f9-9beb-4ea8-9c04-03e02af83a09 |      | hoici | 2018/03/07 14:27:25 | 2018/03/07 14:27:49 | stopped | success | Publish     |
4b6daf36-93b2-4da7-ba47-fcb1a768bca9 |      | hoici | 2018/03/07 14:26:26 | 2018/03/07 14:26:48 | stopped | success | Publish     |
cfa45005-e754-4d3f-b69c-a49fc519190b |      | hoici | 2018/03/07 14:25:16 | 2018/03/07 14:25:42 | stopped | success | Publish     |
958845ac-dada-41e2-a75a-5be10da9e920 |      | hoici | 2018/03/07 14:24:05 | 2018/03/07 14:24:33 | stopped | success | Publish     |
ac135d0d-1c62-421f-b188-c3c81077f580 |      | hoici | 2018/03/07 14:23:05 | 2018/03/07 14:23:29 | stopped | success | Publish     |
1225d280-92fe-4c93-8378-daab860bbc5a |      | hoici | 2018/03/07 14:22:05 | 2018/03/07 14:22:29 | stopped | success | Publish     |
04664f38-13b1-4504-b741-377f09495623 |      | hoici | 2018/03/07 14:20:56 | 2018/03/07 14:21:19 | stopped | success | Publish     |
7f97552d-dc72-4ffb-9a60-7b838a6d977d |      | hoici | 2018/03/07 14:19:47 | 2018/03/07 14:20:11 | stopped | success | Publish     |
82894a59-dd03-4b99-97b9-26a08d4682ec |      | hoici | 2018/03/07 14:18:37 | 2018/03/07 14:19:02 | stopped | success | Publish     |
1927f503-b53c-4652-9266-198f543e2c57 |      | hoici | 2018/03/07 14:17:27 | 2018/03/07 14:17:52 | stopped | success | Publish     |
0e3d6e38-b90a-49c8-bff7-85242f33db33 |      | hoici | 2018/03/07 14:16:28 | 2018/03/07 14:16:50 | stopped | success | Publish     |
5247f5b4-ff03-4ee0-be38-ab02a3e916f4 |      | hoici | 2018/03/07 14:15:30 | 2018/03/07 14:15:51 | stopped | success | Publish     |
7129ce3c-cb1b-4981-8ed6-1a9ee5effe18 |      | hoici | 2018/03/07 14:14:31 | 2018/03/07 14:14:55 | stopped | success | Publish     |
ac7d79c0-805e-47bd-ab21-cc835ef404f2 |      | hoici | 2018/03/07 14:13:45 | 2018/03/07 14:14:02 | stopped | success | Publish     |
d4a757c8-a79b-4845-836b-3bbe314de13c |      | hoici | 2018/03/07 14:13:33 | 2018/03/07 14:13:57 | stopped | success | Publish     |
84a60df0-87b2-415c-a875-c177392c21ea |      | hoici | 2018/03/07 14:12:58 | 2018/03/07 14:13:21 | stopped | success | Publish     |
ddb2faa4-c028-49d0-b613-dc38176faede |      | hoici | 2018/03/07 14:11:34 | 2018/03/07 14:12:09 | stopped | success | Publish     |
4020c634-5763-47bc-a180-a2bef37d17f0 |      | hoici | 2018/03/07 14:09:06 | 2018/03/07 14:09:41 | stopped | success | Publish     |
2ae98279-2c81-4374-a19b-275f07288c1e |      | hoici | 2018/03/07 13:59:22 | 2018/03/07 14:06:36 | stopped | success | Publish     |
b79131db-903f-494a-8f78-b0501f7256f1 |      | hoici | 2018/03/07 13:49:52 | 2018/03/07 13:57:13 | stopped | success | Publish     |
eafbd5ee-c491-4f73-aa20-29a596fe12d4 |      | hoici | 2018/03/07 13:40:31 | 2018/03/07 13:47:47 | stopped | success | Publish     |
11602fd8-673f-4d3f-8dff-53a8d0ad60b2 |      | hoici | 2018/03/07 13:31:07 | 2018/03/07 13:38:26 | stopped | success | Publish     |
1c6c332a-35cb-46e9-81c4-47c774571c47 |      | hoici | 2018/03/07 13:21:45 | 2018/03/07 13:29:00 | stopped | success | Publish     |
4969c8fc-c657-41b5-a4d2-743749e5bdcf |      | hoici | 2018/03/07 13:12:23 | 2018/03/07 13:19:37 | stopped | success | Publish     |
e4adaf9f-96fc-458f-b37c-9c65e9d26c2d |      | hoici | 2018/03/07 13:03:54 | 2018/03/07 13:10:03 | stopped | success | Publish     |
e90329e2-c0d2-4a9f-a235-53441174ef9b |      | hoici | 2018/03/07 12:55:21 | 2018/03/07 13:01:45 | stopped | success | Publish     |
4db2257f-6d55-44bf-91c2-0becdc5d8b3f |      | hoici | 2018/03/06 20:54:00 | 2018/03/06 21:02:40 | stopped | success | Publish     |
d46e9b23-fce9-4310-9832-554900079b2d |      | hoici | 2018/03/06 20:43:26 | 2018/03/06 20:52:06 | stopped | success | Publish     |
72041d50-529b-4672-ae17-ae5588dde681 |      | hoici | 2018/03/06 20:41:29 | 2018/03/06 20:41:44 | stopped | success | Publish     |
68b8e279-064f-431d-a532-196cd48c0a8a |      | hoici | 2018/03/06 20:39:19 | 2018/03/06 20:39:34 | stopped | success | Publish     |
f5ed9197-9dc3-47df-b6b5-5ccafb2ecb0f |      | hoici | 2018/03/06 20:37:09 | 2018/03/06 20:37:26 | stopped | success | Publish     |
0a685208-1b63-4da1-b853-5c5d45fa5da1 |      | hoici | 2018/03/06 20:34:49 | 2018/03/06 20:35:12 | stopped | success | Publish     |
edeaedf9-2494-4fbb-ac72-1f2d1362fa6c |      | hoici | 2018/03/06 20:32:03 | 2018/03/06 20:32:21 | stopped | success | Publish     |
5d83de4a-80d9-4227-906b-92bd92d79785 |      | hoici | 2018/03/06 20:29:34 | 2018/03/06 20:29:53 | stopped | success | Publish     |
dc753009-cad6-4e42-b8e2-468f8bd005d5 |      | hoici | 2018/03/06 20:27:05 | 2018/03/06 20:27:23 | stopped | success | Publish     |
9cec0c13-8760-40d4-bc0e-ab378e4ea0b5 |      | hoici | 2018/03/06 20:24:35 | 2018/03/06 20:24:55 | stopped | success | Publish     |
80c0f06c-94a4-4b46-811f-036fdb9fbb5f |      | hoici | 2018/03/06 20:21:36 | 2018/03/06 20:21:54 | stopped | success | Publish     |
64975eba-c9c3-4a8c-84bf-2d6f009264fa |      | hoici | 2018/03/06 20:18:46 | 2018/03/06 20:19:23 | stopped | success | Publish     |
ba5cc73a-fa6e-4e3d-8aa3-40911117dd3d |      | hoici | 2018/03/06 20:15:52 | 2018/03/06 20:16:06 | stopped | success | Publish     |
5cdd901c-8187-4933-ad21-cc4efca1e5ef |      | hoici | 2018/03/06 20:13:51 | 2018/03/06 20:14:04 | stopped | success | Publish     |
70e5071e-3580-47ed-bb19-38b4ad977dc5 |      | hoici | 2018/03/06 20:11:17 | 2018/03/06 20:11:32 | stopped | success | Publish     |
a6867bda-e4b2-4c8b-91da-73626b93554d |      | hoici | 2018/03/06 20:09:15 | 2018/03/06 20:09:32 | stopped | success | Publish     |
bfe989dd-09de-4677-a376-069ef75c034b |      | hoici | 2018/03/06 20:07:12 | 2018/03/06 20:07:26 | stopped | success | Publish     |
ec206157-f0ca-40d5-bc1c-1a7299855401 |      | hoici | 2018/03/06 20:05:19 | 2018/03/06 20:05:32 | stopped | success | Publish     |
9db2a361-1748-4482-ae6e-fccfca192cd7 |      | hoici | 2018/03/06 20:03:27 | 2018/03/06 20:03:41 | stopped | success | Publish     |
77156464-9678-4778-b83d-83cc7372ab37 |      | hoici | 2018/03/06 20:01:36 | 2018/03/06 20:01:52 | stopped | success | Publish     |
-------------------------------------|------|-------|---------------------|---------------------|---------|---------|-------------|------------
[Unknown Hilti Tier] root@li-lc-1578:~# sudo -u hoici hammer -c /opt/hoici/etc/sat6/hammer-hoici.yaml task list --search 'state == stopped and label == Actions::Katello::ContentView::Promote'
-------------------------------------|------|-------|---------------------|---------------------|---------|---------|-------------|------------
ID                                   | NAME | OWNER | STARTED AT          | ENDED AT            | STATE   | RESULT  | TASK ACTION | TASK ERRORS
-------------------------------------|------|-------|---------------------|---------------------|---------|---------|-------------|------------
f38166e0-23bd-49f4-bc91-7712abb2f755 |      | hoici | 2018/03/07 14:32:02 | 2018/03/07 14:32:17 | stopped | success | Promote     |
7cf55dbe-e03e-4862-b98e-07702ec207b3 |      | hoici | 2018/03/07 14:31:02 | 2018/03/07 14:31:17 | stopped | success | Promote     |
c45ed39f-3839-40f5-ab6d-0928090e563f |      | hoici | 2018/03/07 14:30:01 | 2018/03/07 14:30:17 | stopped | success | Promote     |
7a1d7dc5-ec61-485f-8299-5a79d18d499e |      | hoici | 2018/03/07 14:29:01 | 2018/03/07 14:29:16 | stopped | success | Promote     |
2ae3e177-bf2e-4193-bb30-ce530827cf37 |      | hoici | 2018/03/07 14:28:02 | 2018/03/07 14:28:19 | stopped | success | Promote     |
4b203bce-680a-4012-b47f-a3752580bbde |      | hoici | 2018/03/07 14:26:53 | 2018/03/07 14:27:09 | stopped | success | Promote     |
6ebba4cd-64aa-41c1-ae2c-db8733526cda |      | hoici | 2018/03/07 14:25:54 | 2018/03/07 14:26:12 | stopped | success | Promote     |
a8f01d6e-f00c-4ea0-9f29-347a7f9ea131 |      | hoici | 2018/03/07 14:24:44 | 2018/03/07 14:25:00 | stopped | success | Promote     |
9a459b3d-9cce-4d09-9f2f-d28d59c7fcc9 |      | hoici | 2018/03/07 14:23:33 | 2018/03/07 14:23:50 | stopped | success | Promote     |
f330d3d0-ab6e-4018-ac27-c27bd52c33cc |      | hoici | 2018/03/07 14:22:33 | 2018/03/07 14:22:49 | stopped | success | Promote     |
f6a9beca-a5ea-4fd5-9e62-3b11a3d21950 |      | hoici | 2018/03/07 14:21:33 | 2018/03/07 14:21:50 | stopped | success | Promote     |
f1fec4f8-0165-486f-86cd-517b22a9ef83 |      | hoici | 2018/03/07 14:20:24 | 2018/03/07 14:20:39 | stopped | success | Promote     |
14812048-889d-4c20-a3a5-26f0b828c4e1 |      | hoici | 2018/03/07 14:19:15 | 2018/03/07 14:19:32 | stopped | success | Promote     |
5fa20593-163e-465f-8a3e-7e1430652f04 |      | hoici | 2018/03/07 14:18:05 | 2018/03/07 14:18:22 | stopped | success | Promote     |
53ca96c7-18a8-4fc3-99c2-179a97f34b35 |      | hoici | 2018/03/07 14:16:55 | 2018/03/07 14:17:12 | stopped | success | Promote     |
683aa7ae-9f57-4659-8666-d643deafaca9 |      | hoici | 2018/03/07 14:15:57 | 2018/03/07 14:16:12 | stopped | success | Promote     |
6904a1d7-2c3e-4217-9881-2983652a691c |      | hoici | 2018/03/07 14:14:58 | 2018/03/07 14:15:14 | stopped | success | Promote     |
d5b4dc93-6fd1-473c-a30a-c79d8a13ad3b |      | hoici | 2018/03/07 14:14:00 | 2018/03/07 14:14:15 | stopped | success | Promote     |
-------------------------------------|------|-------|---------------------|---------------------|---------|---------|-------------|------------
------

Many of the above were CompositeContentViews with 40.000+ packages each
Comment 14 Peter Vreman 2018-03-07 18:33:20 UTC 
Below summary of a non-Lab customer instance average publish/promotion times.

Both Sat63 and Sat62 have as same content assinged
Sat63 is based on 50 published and 20 promotions
Sat62 averages are based on 250 publishes and 150 promotions spread over 30days

Sat63 Publish			00:01:42
Sat62 PublishMax1Hour		00:06:01
Sat62 PublishAll		00:15:53   (includes here are 25 publishes that took between 1-3 hours)
			
Sat63Promote			00:00:16
Sat62Promote			00:04:37
Comment 16 jcallaha 2018-03-28 19:09:38 UTC 
Verified in Satellite 6.3.1 Snap 1.

This bug performance-focused.

The most notable improvements have been to content view promotion as well as composite content view publishing and promotion. Below is the publish of a large composite content view.

Composite:
     6 Content Views
    15 Repositories
43,503 Packages
 3,483 Errata

Action Type   Publish 
User          admin 
Started At    3/26/18 3:07 PM
Finished At   3/26/18 3:07 PM
Parameters    content view 'all' ; organization 'Default Organization' 
State         stopped
Result        success

Additionally, there have been no identified regressions during our automation runs.
Comment 18 errata-xmlrpc 2018-04-13 13:29:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1126