Bug 1997623 - Capsule sync fails with duplicate key and deadlock
Summary: Capsule sync fails with duplicate key and deadlock
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.10.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: 6.10.0
Assignee: satellite6-bugs
QA Contact: Danny Synk
URL:
Whiteboard:
: 1996480 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-25 14:44 UTC by Kenny Tordeurs
Modified: 2021-11-16 14:13 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-16 14:13:37 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 9379 0 High CLOSED - CURRENTRELEASE Backport "Deadlock on rpm repository pulp2pulp sync"to 3.14.z 2021-09-29 19:07:06 UTC
Pulp Redmine 9445 0 High CLOSED - CURRENTRELEASE Backport request to 3.14.z: deadlocks from touch() in several paths 2021-09-29 19:06:54 UTC
Red Hat Product Errata RHSA-2021:4702 0 None None None 2021-11-16 14:13:45 UTC

Description Kenny Tordeurs 2021-08-25 14:44:47 UTC
Description of problem:
Capsule sync (optimized sync) fails upon first sync after capsule was installed.

Version-Release number of selected component (if applicable):
satellite-capsule-6.10.0-0.6.beta.el7sat.noarch

How reproducible:
Unsure

Steps to Reproduce:
1. Installed capsule 6.10
2. Launched a sync
3.

Actual results:
~~~
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(Server, Server, Server, variant, Packages, 00282cb5-53fb-4e49-b1db-ccf9ffdf7e17) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(Server, Server, Server, variant, Packages, 7655b202-cabd-4740-ac09-0a892125e7e1) already exists.
deadlock detected
DETAIL:  Process 9122 waits for ShareLock on transaction 579218; blocked by process 12902.
Process 12902 waits for ShareLock on transaction 578810; blocked by process 9122.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (30507,3) in relation "rpm_package_pkgId_key"
~~~

Expected results:
no errors

Additional info:

Comment 2 Kenny Tordeurs 2021-08-25 15:00:24 UTC
production.log has the following traceback:

2021-08-25T15:46:53 [E|bac|f1534055] deadlock detected
 f1534055 | DETAIL:  Process 9122 waits for ShareLock on transaction 579218; blocked by process 12902.
 f1534055 | Process 12902 waits for ShareLock on transaction 578810; blocked by process 9122.
 f1534055 | HINT:  See server log for query details.
 f1534055 | CONTEXT:  while inserting index tuple (30507,3) in relation "rpm_package_pkgId_key"
 f1534055 |  (Katello::Errors::Pulp3Error)
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/polling.rb:22:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/cancellable.rb:14:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:572:in `block (3 levels) in execute_run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/middleware/record_smart_proxy_sync_history.rb:26:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:32:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/middleware/remote_action.rb:16:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/progress.rb:17:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.3/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.3/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.3/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/world.rb:31:in `execute'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:571:in `block (2 levels) in execute_run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:570:in `catch'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:570:in `block in execute_run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in `block in with_error_handling'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in `catch'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in `with_error_handling'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:565:in `execute_run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:286:in `execute'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/director.rb:94:in `execute'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors.rb:18:in `run_user_code'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 f1534055 | [ sidekiq ]
 f1534055 | [ concurrent-ruby ]

Comment 3 Brad Buckingham 2021-08-30 12:24:41 UTC
Bug 1974482 was opened and closed as not reproducible.  Based upon this one, it appears the issue does still exist.

Another related one may be bug 1996480.

Comment 4 pulp-infra@redhat.com 2021-09-10 18:08:59 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2021-09-10 18:09:00 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 6 Grant Gainey 2021-09-21 13:34:21 UTC
*** Bug 1996480 has been marked as a duplicate of this bug. ***

Comment 7 pulp-infra@redhat.com 2021-09-24 13:15:13 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 8 pulp-infra@redhat.com 2021-09-24 13:15:14 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 9 pulp-infra@redhat.com 2021-09-28 16:11:21 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 10 pulp-infra@redhat.com 2021-09-28 21:06:09 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 11 pulp-infra@redhat.com 2021-09-29 02:17:51 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 12 pulp-infra@redhat.com 2021-09-29 04:10:23 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 13 pulp-infra@redhat.com 2021-09-29 05:07:23 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 14 pulp-infra@redhat.com 2021-09-29 19:06:55 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 15 pulp-infra@redhat.com 2021-09-29 19:07:07 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 16 Danny Synk 2021-10-04 15:19:40 UTC
Verified on Satellite 6.10, snap 21.

Steps to Test:
1. Deploy a Satellite 6.10 and a Capsule 6.10.
2. Configure the Capsule with a default SSL certificate from the Satellite.
3. Create a new lifecycle environment on the Satellite.
4. Synchronize one repository to the Satellite.
5. Create a new content view, add the repository synced in step 4, publish the first version of the content view, and promote the version to the lifecycle environment created in step 3.
5. In the Satellite webUI, on the Infrastructure > Capsules > [Capsule name] > Edit > Lifecycle Environments tab, add the lifecycle environment created in step 3 to the Capsule.
6. On the Infrastructure > Capsules > [Capsule name] page, select "Optimized Sync" from the "Synchronize" dropdown menu.

Expected Results:
The Capsule synchronizes successfully with no errors.

Actual Results:
The Capsule synchronizes successfully with no error.

Comment 19 errata-xmlrpc 2021-11-16 14:13:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4702


Note You need to log in before you can comment on or make changes to this bug.