Bug 1425856

Summary: pulp celery worker consumes huge memory during Content View publish , due to missing fields in copying errata
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: Content ViewsAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Ales Dujicek <adujicek>
Severity: medium Docs Contact:
Priority: high    
Version: 6.2.8CC: abalakht, adujicek, bbuckingham, egolov, fgarciad, jcallaha, jsherril, ktordeur, mverma
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-21 16:46:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
free memory none

Description Pavel Moravec 2017-02-22 15:40:52 UTC 
Description of problem:
Having a large repo with many (>4000) errata added to a Content View, then publishing this C.V. causes pulp celery worker consuming much memory (i.e. 5GB). Reasoning is described in 

http://pulpproject.org/2017/02/17/why-does-copy-use-lots-of-memory/

Let specify fields uniquely identifying errata like katello does for RPM in:

https://github.com/Katello/katello/blob/master/app/services/katello/pulp/rpm.rb#L6


Version-Release number of selected component (if applicable):
Sat 6.2.7


How reproducible:
100%


Steps to Reproduce:
1. Sync a big repo with >4k errata
2. Create a Content View with that repo
3. Publish the repo
4. During the publish, monitor memory usage of pulp celery workers


Actual results:
4. shows very high memory usage (seen 5GB, depends a lot on repo)


Expected results:
4. shows reasonable low memory usage


Additional info:
Comment 6 Pavel Moravec 2017-03-15 12:10:53 UTC 
Perspective patch of 6.2.8:

--- /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.95/app/services/katello/pulp/erratum.rb	2017-03-14 17:03:36.959539640 -0400
+++ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.95/app/services/katello/pulp/erratum.rb.new	2017-03-14 17:01:08.567562162 -0400
@@ -1,6 +1,7 @@
 module Katello
   module Pulp
     class Erratum < PulpContentUnit
+      PULP_ERRATA_SELECT_FIELDS = %w(id)
       CONTENT_TYPE = "erratum"
 
       def self.unit_handler
--- /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.95/app/lib/actions/pulp/repository/copy_errata.rb	2017-03-14 17:03:37.474539547 -0400
+++ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.95/app/lib/actions/pulp/repository/copy_errata.rb.new	2017-03-14 17:01:12.093562059 -0400
@@ -5,6 +5,10 @@ module Actions
         def content_extension
           pulp_extensions.errata
         end
+
+        def criteria
+          super.merge(fields: ::Katello::Pulp::Erratum::PULP_ERRATA_SELECT_FIELDS)
+        end
       end
     end
   end
--- /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.95/app/lib/actions/pulp/repository/remove_errata.rb	2017-03-14 17:03:37.954539460 -0400
+++ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.95/app/lib/actions/pulp/repository/remove_errata.rb.new	2017-03-14 17:01:16.422561931 -0400
@@ -5,6 +5,10 @@ module Actions
         def content_extension
           pulp_extensions.errata
         end
+
+        def criteria
+          super.merge(fields: { :unit => ::Katello::Pulp::Erratum::PULP_ERRATA_SELECT_FIELDS})
+        end
       end
     end
   end
--- /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.95/app/models/katello/glue/pulp/repo.rb	2017-03-14 17:03:38.455539369 -0400
+++ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.95/app/models/katello/glue/pulp/repo.rb.new	2017-03-14 17:04:32.317529632 -0400
@@ -413,8 +413,11 @@ module Katello
         fail "Invalid content type #{content_type} sent. It needs to be one of #{content_classes.keys}"\
                                                                        unless content_classes[content_type]
         criteria = {}
-        if content_type == Runcible::Extensions::Rpm.content_type
+        case content_type
+        when Runcible::Extensions::Rpm.content_type
           criteria[:fields] = Pulp::Rpm::PULP_SELECT_FIELDS
+        when Runcible::Extensions::Errata.content_type
+          criteria[:fields] = Pulp::Erratum::PULP_ERRATA_SELECT_FIELDS
         end
 
         if filter_clauses && !filter_clauses.empty?
@@ -443,7 +446,9 @@ module Katello
 
           # Since the rpms will be copied above, during the copy of errata and package groups,
           # include the copy_children flag to request that pulp skip copying them again.
-          events << Katello.pulp_server.extensions.errata.copy(self.pulp_id, to_repo.pulp_id,  :copy_children => false)
+          events << Katello.pulp_server.extensions.errata.copy(self.pulp_id, to_repo.pulp_id,
+                                                    :fields => Pulp::Erratum::PULP_ERRATA_SELECT_FIELDS,
+                                                    :copy_children => false)
           events << Katello.pulp_server.extensions.package_group.copy(self.pulp_id, to_repo.pulp_id,  :copy_children => false)
           events << clone_file_metadata(to_repo)
         end
@@ -457,8 +462,11 @@ module Katello
 
       def unassociate_by_filter(content_type, filter_clauses)
         criteria = {:type_ids => [content_type], :filters => {:unit => filter_clauses}}
-        if content_type == Katello.pulp_server.extensions.rpm.content_type
+        case content_type
+        when Katello.pulp_server.extensions.rpm.content_type
           criteria[:fields] = { :unit => Pulp::Rpm::PULP_SELECT_FIELDS}
+        when Katello.pulp_server.extensions.errata.content_type
+          criteria[:fields] = { :unit => Pulp::Erratum::PULP_ERRATA_SELECT_FIELDS}
         end
         Katello.pulp_server.extensions.repository.unassociate_units(self.pulp_id, criteria)
       end


Put it to /tmp/bz1425856.patch, then run:

cat /tmp/bz1425856.patch | patch -p1

"katello-service restart" to apply it.

Anyway, this patch is so far unofficial..
Comment 7 Pavel Moravec 2017-03-20 16:04:45 UTC 
POSTed to upstream via:

http://projects.theforeman.org/projects/katello/repository/katello/revisions/c83af8691f238ac106b416b8574140d069342295

upstream bug closed, dunno why the change isnt propagated here.
Comment 8 Satellite Program 2017-03-20 16:10:08 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/18916 has been resolved.
Comment 10 Ales Dujicek 2018-01-03 12:10:04 UTC 
Created attachment 1376301 [details]
free memory

free memory during verification
Comment 11 Bryan Kearney 2018-02-21 16:42:12 UTC 
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336
Comment 12 Bryan Kearney 2018-02-21 16:46:14 UTC 
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336
Comment 13 Satellite Program 2018-02-21 16:54:37 UTC 
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> > 
> > For information on the advisory, and where to find the updated files, follow the link below.
> > 
> > If the solution does not work for you, open a new bug report.
> > 
> > https://access.redhat.com/errata/RHSA-2018:0336