Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2077363

Summary: Fail to sync kickstart repositories with same sub repositories concurrently
Product: Red Hat Satellite Reporter: Hao Chang Yu <hyu>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Vladimír Sedmík <vsedmik>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.10.0CC: dalley, dkliban, dprabhu, fhirtz, ggainey, hartsjc, jalviso, jangerrit.kootstra, jpasqual, matthew.lesieur, musman, pcreech, pdwyer, rchan, risantam, sadas, satellite6-bugs, vogt, wclark
Target Milestone: 6.13.0Keywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pulp-rpm-3.18.9-1 pulp-rpm-3.17.18-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-05-03 13:21:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Hotfix RPM for Satellite 6.11.5 on RHEL7
none
Hotfix RPM for Satellite 6.11.5 on RHEL8 none

Description Hao Chang Yu 2022-04-21 07:45:20 UTC 
Description of problem:
Capsule sync failed with the following errors when syncing a number of kickstart repositories sharing the same sub repositories concurrently.

--------------------------------------------------------------
duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_3c54ce50_uniq"
DETAIL:  Key (repository_id, number)=(0b5936ef-2971-459f-90e2-9d1a1d837292, 1) already exists.
Could not lookup a publication_href for repo 176deadlock detected

DETAIL:  Process 7734 waits for ShareLock on transaction 45647574; blocked by process 7737.
Process 7737 waits for ShareLock on transaction 45647572; blocked by process 7734.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (1067,1) in relation "core_artifact_sha256_key"
Could not lookup a publication_href for repo 177

insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL:  Key (version_added_id)=(5d344d1d-76c9-476c-980a-11a240881389) is not present in table "core_repositoryversion".
Could not lookup a publication_href for repo 181

insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL:  Key (version_added_id)=(18077abc-2cba-40db-a36e-f4dda3142057) is not present in table "core_repositoryversion".
Could not lookup a publication_href for repo 184

deadlock detected
DETAIL:  Process 7766 waits for ShareLock on transaction 45647572; blocked by process 7734.
Process 7734 waits for ShareLock on transaction 45647574; blocked by process 7737.
Process 7737 waits for ShareLock on transaction 45647571; blocked by process 7766.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (1066,3) in relation "core_artifact_sha256_key"
Could not lookup a publication_href for repo 186
--------------------------------------------------------------


Error details and tracebacks:

Error #1:
---------------------------------------------------------------
pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/93bde0ef-cc33-434f-9922-4dcacd482039/"
  pulp_created: '2022-04-21T07:01:30.908+00:00'
  state: failed
  name: pulp_rpm.app.tasks.synchronizing.synchronize
  logging_cid: dcd8227c-bfa0-4400-8d97-749f5f2b7fcd
  started_at: '2022-04-21T07:01:31.114+00:00'
  finished_at: '2022-04-21T07:01:32.726+00:00'
  error:
    traceback: |2
        File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 339, in _perform_task
          result = func(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 474, in synchronize
          subrepo_version = dv.create()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 145, in create
          with self.repository.new_version() as new_version:
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py", line 258, in new_version
          version.save()
        File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 134, in save
          save(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 744, in save
          force_update=force_update, update_fields=update_fields)
        File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 782, in save_base
          force_update, using, update_fields,
        File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 873, in _save_table
          result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
        File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 911, in _do_insert
          using=using, raw=raw)
        File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
          return getattr(self.get_queryset(), name)(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
          return query.get_compiler(using=using).execute_sql(return_id)
        File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
          cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
          return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
          return executor(sql, params, many, context)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
          raise dj_exc_value.with_traceback(traceback) from exc_value
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
    description: |
      duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_3c54ce50_uniq"
      DETAIL:  Key (repository_id, number)=(0b5936ef-2971-459f-90e2-9d1a1d837292, 1) already exists.
  worker: "/pulp/api/v3/workers/2482a7ef-daa8-4fd2-af66-3cadced8c991/"
---------------------------------------------------------------


# Error #2
--------------------------------------------------------------
pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/b6a1e67b-3d5e-4737-bc93-72172f72b97a/"
  pulp_created: '2022-04-21T07:01:30.782+00:00'
  state: failed
  name: pulp_rpm.app.tasks.synchronizing.synchronize
  logging_cid: dcd8227c-bfa0-4400-8d97-749f5f2b7fcd
  started_at: '2022-04-21T07:01:31.090+00:00'
  finished_at: '2022-04-21T07:02:13.061+00:00'
  error:
    traceback: |2
        File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 339, in _perform_task
          result = func(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 474, in synchronize
          subrepo_version = dv.create()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
          loop.run_until_complete(pipeline)
        File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
          return future.result()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
          await asyncio.gather(*futures)
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
          await self.run()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py", line 287, in run
          self.new_version.add_content(Content.objects.filter(pk__in=to_add))
        File "/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 764, in add_content
          RepositoryContent.objects.bulk_create(repo_content)
        File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
          return getattr(self.get_queryset(), name)(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 480, in bulk_create
          obj_without_pk._state.db = self.db
        File "/usr/lib/python3.6/site-packages/django/db/transaction.py", line 240, in __exit__
          connection.commit()
        File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 262, in commit
          self._commit()
        File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
          return self.connection.commit()
        File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
          raise dj_exc_value.with_traceback(traceback) from exc_value
        File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
          return self.connection.commit()
    description: |
      insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
      DETAIL:  Key (version_added_id)=(5d344d1d-76c9-476c-980a-11a240881389) is not present in table "core_repositoryversion".
---------------------------------------------------------------


Error #3: Should be fixed by bug #2062526
---------------------------------------------------------------
pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/eff5f426-baee-4334-8cfa-869e527ddf01/"
  pulp_created: '2022-04-21T07:01:30.792+00:00'
  state: failed
  name: pulp_rpm.app.tasks.synchronizing.synchronize
  logging_cid: dcd8227c-bfa0-4400-8d97-749f5f2b7fcd
  started_at: '2022-04-21T07:01:31.055+00:00'
  finished_at: '2022-04-21T07:02:11.478+00:00'
  error:
    traceback: |2
        File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 339, in _perform_task
          result = func(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 474, in synchronize
          subrepo_version = dv.create()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
          loop.run_until_complete(pipeline)
        File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
          return future.result()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
          await asyncio.gather(*futures)
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
          await self.run()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 241, in run
          d_artifact.artifact for d_artifact in da_to_save
        File "/usr/lib/python3.6/site-packages/pulpcore/app/models/content.py", line 84, in bulk_get_or_create
          return super().bulk_create(objs, batch_size=batch_size)
        File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
          return getattr(self.get_queryset(), name)(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 468, in bulk_create
          self._batched_insert(objs_with_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1204, in _batched_insert
          ignore_conflicts=ignore_conflicts,
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
          return query.get_compiler(using=using).execute_sql(return_id)
        File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
          cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
          return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
          return executor(sql, params, many, context)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
          raise dj_exc_value.with_traceback(traceback) from exc_value
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
    description: |
      deadlock detected
      DETAIL:  Process 7734 waits for ShareLock on transaction 45647574; blocked by process 7737.
      Process 7737 waits for ShareLock on transaction 45647572; blocked by process 7734.
      HINT:  See server log for query details.
      CONTEXT:  while inserting index tuple (1067,1) in relation "core_artifact_sha256_key"




How reproducible:
Easy


Steps to Reproduce:
1. Enable and sync a RHEL 6.10 kickstart repository.
2. Create a content view and attach the RHEL 6.10 kickstart repository
3. Clone 7 copied of the content view
4. Disable Capsule auto-sync setting
5. Publish and promote the content views to a lifecycle environment.
6. Trigger an optimized Capsule sync manually.

Actual results:
See errors above

Expected results:
No error

Additional info:
I think it is because multiple kickstart repositories shared the same sub repositories. When syncing those repositories concurrently, race conditions occur trying to update the same sub repositories.


# select pulp_id,  name, next_version, pulp_type, retained_versions from core_repository where name like '%e82fa8cdfb0341557859910c7f96773b9b307e760d83f06ba93432c783c70cb';
               pulp_id                |                                        name                                         | next_version | pulp_type | retained_versions 
--------------------------------------+-------------------------------------------------------------------------------------+--------------+-----------+-------------------
 b8a0d4ed-3e90-4f8b-a95b-45c436e4c6e5 | ResilientStorage-be82fa8cdfb0341557859910c7f96773b9b307e760d83f06ba93432c783c70cb   |            2 | rpm.rpm   |                  
 ddc720ef-0e59-468f-a64e-e6dd7fb210c2 | Server-be82fa8cdfb0341557859910c7f96773b9b307e760d83f06ba93432c783c70cb             |            2 | rpm.rpm   |                  
 ba50f4fe-3bd3-44cf-a55a-0b00da0bba70 | HighAvailability-be82fa8cdfb0341557859910c7f96773b9b307e760d83f06ba93432c783c70cb   |            2 | rpm.rpm   |                  
 aafac7c5-0fd6-4f54-a34e-9bba69b39259 | ScalableFileSystem-be82fa8cdfb0341557859910c7f96773b9b307e760d83f06ba93432c783c70cb |            2 | rpm.rpm   |                  
 ec35546c-eaa7-4577-a1bc-0eefc1f6bbf9 | LoadBalancer-be82fa8cdfb0341557859910c7f96773b9b307e760d83f06ba93432c783c70cb       |            2 | rpm.rpm   |                  
(5 rows)


In /usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py
---------------------------------------------------------------
...
    if treeinfo:
        treeinfo["repositories"] = {}
        for repodata in set(treeinfo["download"]["repodatas"]):
            if repodata == DIST_TREE_MAIN_REPO_PATH:
                treeinfo["repositories"].update({repodata: None})
                continue
            name = f"{repodata}-{treeinfo['hash']}"  <==========
            sub_repo, created = RpmRepository.objects.get_or_create(name=name, sub_repo=True)  <===========
            if created:
                sub_repo.save()
            directory = treeinfo["repo_map"][repodata]
            treeinfo["repositories"].update({directory: str(sub_repo.pk)})
            ...
---------------------------------------------------------------


My guess for the Error #2:

There is a code to delete an incomplete latest version before creating a new version so I think below is the possible scenario.

1) 2 tasks are processing the same sub repository.
2) Task 'A' creates a latest version and proceed but not complete yet.
3) Task 'B' deletes the version created by Task 'A' and creates another version.
4) Task 'A' tries to associate content to repo then fails.


In /usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py
---------------------------------------------------------------
    def new_version(self, base_version=None):
        """
        Create a new RepositoryVersion for this Repository

        Creation of a RepositoryVersion should be done in a RQ Job.

        Args:
            repository (pulpcore.app.models.Repository): to create a new version of
            base_version (pulpcore.app.models.RepositoryVersion): an optional repository version
                whose content will be used as the set of content for the new version

        Returns:
            pulpcore.app.models.RepositoryVersion: The Created RepositoryVersion
        """
        with transaction.atomic():
            latest_version = self.versions.latest()
            if not latest_version.complete:  <=====================
                latest_version.delete()
---------------------------------------------------------------
Comment 4 Daniel Alley 2022-11-04 23:44:56 UTC 
*** Bug 2103246 has been marked as a duplicate of this bug. ***
Comment 6 pulp-infra@redhat.com 2022-11-16 14:07:21 UTC 
The Pulp upstream bug status is at closed. Updating the external tracker on this bug.
Comment 7 Daniel Alley 2022-11-21 15:50:12 UTC 
It would be good to see this go into 6.12.1
Comment 12 ir. Jan Gerrit Kootstra 2023-01-24 07:49:51 UTC 
On 6.12.0 we still have the same issue.
Have not been able to upgrade to 6.12.1
https://access.redhat.com/solutions/6983687 causes performance issue, so we are running out of the change window of 8 hours (in the past we needed only 4 hours, but due to the size of RHEL 6.10, RHEL 7.9, RHEL 8.7 and RHEL 9.1 content we already need 7 hours if we do not have issues).
Comment 14 Vladimír Sedmík 2023-02-17 21:17:17 UTC 
Verified in 6.13.0 snap 10 (with python39-pulp-rpm-3.18.9-1.el8pc.noarch)

Following the steps in comment#0 no error was observed during the capsule sync.

Tested also with different KS repos (rhel6, rhel7, rhel8-baseOS, rhel8-apps, rhel9-baseOS, rhel9-apps), download policies on the SAT side (on_demand/immediate), more CVs count.
Comment 22 errata-xmlrpc 2023-05-03 13:21:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.13 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2097
Comment 31 wclark 2023-05-22 23:55:00 UTC 
Created attachment 1966315 [details]
Hotfix RPM for Satellite 6.11.5 on RHEL7

A Hotfix RPM for Satellite 6.11.5 on RHEL7 is now available.

INSTALL INSTRUCTIONS:

1. Take a complete backup or snapshot of the Satellite 6.11.5 server

2. Download the hotfix RPM from this message's attachment.

3. # yum install ./tfm-pulpcore-python3-pulp-rpm-3.17.10-3.HOTFIXRHBZ2077363.el7pc.noarch.rpm --disableplugin=foreman-protector

4. # satellite-maintain service restart
Comment 32 wclark 2023-05-22 23:59:27 UTC 
Created attachment 1966316 [details]
Hotfix RPM for Satellite 6.11.5 on RHEL8

A Hotfix RPM for Satellite 6.11.5 on RHEL8 is now available.

INSTALL INSTRUCTIONS:

1. Take a complete backup or snapshot of the Satellite 6.11.5 server

2. Download the hotfix RPM from this message's attachment.

3. # dnf install ./python38-pulp-rpm-3.17.10-3.HOTFIXRHBZ2077363.el8pc.noarch.rpm --disableplugin=foreman-protector

4. # satellite-maintain service restart
Comment 34 wclark 2023-05-23 18:15:49 UTC 
Correction: This hotfix needs to be deployed on Capsule servers to prevent the originally reported issue of failing Capsule syncs when syncing multiple kickstart repositories
Comment 36 Red Hat Bugzilla 2023-10-26 04:25:05 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days