Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2008602

Summary: Katello::Errors::Pulp3Error: invalid memory alloc request size while syncing third-party repos
Product: Red Hat Satellite Reporter: Ganesh Payelkar <gpayelka>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED NOTABUG QA Contact: Lai <ltran>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.10.0CC: dalley, ggainey, rchan, ttereshc
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-06-07 23:31:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ganesh Payelkar 2021-09-28 16:22:34 UTC
Description of problem:

Katello::Errors::Pulp3Error: invalid memory alloc request size  while syncing third-party repos

Version-Release number of selected component (if applicable):
satellite-6.10.0-0.9.beta

How reproducible:
New installation of satellite 6.10 Beta 

Steps to Reproduce:
1. Install new satellite 6.10 Beta
2. Configure custom product with https://packages.microsoft.com/rhel/8/prod/
3. Start syncing 

Actual results:

Katello::Errors::Pulp3Error: invalid memory alloc request size 1073741824

Expected results:


Additional info:


{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/6142627e-7c88-4617-a3be-0819984ff30d/",
    "pulp_created"=>"2021-09-28T06:07:28.972+00:00",
    "state"=>"failed",
    "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"55bc4cb4-3824-4de3-9e93-25abcbe9f6a5",
    "started_at"=>"2021-09-28T06:07:29.119+00:00",
    "finished_at"=>"2021-09-28T06:21:39.754+00:00",
    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 272, in _perform_task\n" +
       "    result = func(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 475, in synchronize\n" +
       "    version = dv.create()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 151, in create\n" +
       "    loop.run_until_complete(pipeline)\n" +
       "  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n" +
       "    return future.result()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n" +
       "    await asyncio.gather(*futures)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n" +
       "    await self.run()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py\", line 113, in run\n" +
       "    d_content.content.save()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/app/models/base.py\", line 149, in save\n" +
       "    return super().save(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py\", line 134, in save\n" +
       "    save(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/base.py\", line 744, in save\n" +
       "    force_update=force_update, update_fields=update_fields)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/base.py\", line 782, in save_base\n" +
       "    force_update, using, update_fields,\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/base.py\", line 873, in _save_table\n" +
       "    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/base.py\", line 911, in _do_insert\n" +
       "    using=using, raw=raw)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/manager.py\", line 82, in manager_method\n" +
       "    return getattr(self.get_queryset(), name)(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/query.py\", line 1186, in _insert\n" +
       "    return query.get_compiler(using=using).execute_sql(return_id)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py\", line 1377, in execute_sql\n" +
       "    cursor.execute(sql, params)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/backends/utils.py\", line 67, in execute\n" +
       "    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/backends/utils.py\", line 76, in _execute_with_wrappers\n" +
       "    return executor(sql, params, many, context)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute\n" +
       "    return self.cursor.execute(sql, params)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/utils.py\", line 89, in __exit__\n" +
       "    raise dj_exc_value.with_traceback(traceback) from exc_value\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute\n" +
       "    return self.cursor.execute(sql, params)\n",
      "description"=>"invalid memory alloc request size 1073741824\n"},
    "worker"=>"/pulp/api/v3/workers/a938363f-03b0-4940-baa9-4a8be9979658/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Downloading Metadata Files",
       "code"=>"sync.downloading.metadata",
       "state"=>"completed",
       "done"=>8},
      {"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"completed",
       "done"=>154},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"canceled",
       "done"=>0},
      {"message"=>"Parsed Packages",
       "code"=>"sync.parsing.packages",
       "state"=>"completed",
       "total"=>154,
       "done"=>154}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/53932542-5096-4944-a6cc-5a0f3b8f7a5a/",
      "/pulp/api/v3/remotes/rpm/rpm/beb2e274-f21f-4beb-a1f8-9ba517c056bd/"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>52, "failed"=>1}}

Comment 3 pulp-infra@redhat.com 2021-09-30 21:07:15 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 4 pulp-infra@redhat.com 2021-09-30 21:07:16 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 5 Daniel Alley 2021-12-06 00:28:08 UTC
This is not a problem with Pulp / Satellite per se.  This repo seems to contain a set of packages with *millions* of duplicated file entries - the same set of files are listed nearly 13 million times for one particular package and nearly 3 million times for another.

The duplicate file listings cause the metadata to be so inflated that it exceeds the amount of data that can be inserted into Postgresql in a single insert.

I've now reported it upstream to Microsoft and closed the associated Pulp issues.

https://github.com/dotnet/core/issues/6706#issuecomment-986330681

Comment 6 pulp-infra@redhat.com 2021-12-06 00:30:54 UTC
The Pulp upstream bug status is at CLOSED - NOTABUG. Updating the external tracker on this bug.

Comment 7 Daniel Alley 2021-12-17 20:42:16 UTC
As of today it appears that Microsoft has fixed their repository, it should now be able to be synced by Pulp / Satellite.