Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2008602 - Katello::Errors::Pulp3Error: invalid memory alloc request size while syncing third-party repos
Summary: Katello::Errors::Pulp3Error: invalid memory alloc request size while syncing...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.10.0
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Lai
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-09-28 16:22 UTC by Ganesh Payelkar
Modified: 2024-12-20 21:13 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-06-07 23:31:51 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 9399 0 High CLOSED - NOTABUG sync error: invalid memory alloc request size 2021-12-06 00:30:53 UTC

Description Ganesh Payelkar 2021-09-28 16:22:34 UTC
Description of problem:

Katello::Errors::Pulp3Error: invalid memory alloc request size  while syncing third-party repos

Version-Release number of selected component (if applicable):
satellite-6.10.0-0.9.beta

How reproducible:
New installation of satellite 6.10 Beta 

Steps to Reproduce:
1. Install new satellite 6.10 Beta
2. Configure custom product with https://packages.microsoft.com/rhel/8/prod/
3. Start syncing 

Actual results:

Katello::Errors::Pulp3Error: invalid memory alloc request size 1073741824

Expected results:


Additional info:


{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/6142627e-7c88-4617-a3be-0819984ff30d/",
    "pulp_created"=>"2021-09-28T06:07:28.972+00:00",
    "state"=>"failed",
    "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"55bc4cb4-3824-4de3-9e93-25abcbe9f6a5",
    "started_at"=>"2021-09-28T06:07:29.119+00:00",
    "finished_at"=>"2021-09-28T06:21:39.754+00:00",
    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 272, in _perform_task\n" +
       "    result = func(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 475, in synchronize\n" +
       "    version = dv.create()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 151, in create\n" +
       "    loop.run_until_complete(pipeline)\n" +
       "  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n" +
       "    return future.result()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n" +
       "    await asyncio.gather(*futures)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n" +
       "    await self.run()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py\", line 113, in run\n" +
       "    d_content.content.save()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/app/models/base.py\", line 149, in save\n" +
       "    return super().save(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py\", line 134, in save\n" +
       "    save(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/base.py\", line 744, in save\n" +
       "    force_update=force_update, update_fields=update_fields)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/base.py\", line 782, in save_base\n" +
       "    force_update, using, update_fields,\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/base.py\", line 873, in _save_table\n" +
       "    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/base.py\", line 911, in _do_insert\n" +
       "    using=using, raw=raw)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/manager.py\", line 82, in manager_method\n" +
       "    return getattr(self.get_queryset(), name)(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/query.py\", line 1186, in _insert\n" +
       "    return query.get_compiler(using=using).execute_sql(return_id)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py\", line 1377, in execute_sql\n" +
       "    cursor.execute(sql, params)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/backends/utils.py\", line 67, in execute\n" +
       "    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/backends/utils.py\", line 76, in _execute_with_wrappers\n" +
       "    return executor(sql, params, many, context)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute\n" +
       "    return self.cursor.execute(sql, params)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/utils.py\", line 89, in __exit__\n" +
       "    raise dj_exc_value.with_traceback(traceback) from exc_value\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute\n" +
       "    return self.cursor.execute(sql, params)\n",
      "description"=>"invalid memory alloc request size 1073741824\n"},
    "worker"=>"/pulp/api/v3/workers/a938363f-03b0-4940-baa9-4a8be9979658/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Downloading Metadata Files",
       "code"=>"sync.downloading.metadata",
       "state"=>"completed",
       "done"=>8},
      {"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"completed",
       "done"=>154},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"canceled",
       "done"=>0},
      {"message"=>"Parsed Packages",
       "code"=>"sync.parsing.packages",
       "state"=>"completed",
       "total"=>154,
       "done"=>154}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/53932542-5096-4944-a6cc-5a0f3b8f7a5a/",
      "/pulp/api/v3/remotes/rpm/rpm/beb2e274-f21f-4beb-a1f8-9ba517c056bd/"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>52, "failed"=>1}}

Comment 3 pulp-infra@redhat.com 2021-09-30 21:07:15 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 4 pulp-infra@redhat.com 2021-09-30 21:07:16 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 5 Daniel Alley 2021-12-06 00:28:08 UTC
This is not a problem with Pulp / Satellite per se.  This repo seems to contain a set of packages with *millions* of duplicated file entries - the same set of files are listed nearly 13 million times for one particular package and nearly 3 million times for another.

The duplicate file listings cause the metadata to be so inflated that it exceeds the amount of data that can be inserted into Postgresql in a single insert.

I've now reported it upstream to Microsoft and closed the associated Pulp issues.

https://github.com/dotnet/core/issues/6706#issuecomment-986330681

Comment 6 pulp-infra@redhat.com 2021-12-06 00:30:54 UTC
The Pulp upstream bug status is at CLOSED - NOTABUG. Updating the external tracker on this bug.

Comment 7 Daniel Alley 2021-12-17 20:42:16 UTC
As of today it appears that Microsoft has fixed their repository, it should now be able to be synced by Pulp / Satellite.


Note You need to log in before you can comment on or make changes to this bug.