Bug 1997623

Summary: Capsule sync fails with duplicate key and deadlock
Product: Red Hat Satellite Reporter: Kenny Tordeurs <ktordeur>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Danny Synk <dsynk>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.10.0CC: ggainey, rchan, tasander, ttereshc, zhunting
Target Milestone: 6.10.0Keywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-16 14:13:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kenny Tordeurs 2021-08-25 14:44:47 UTC
Description of problem:
Capsule sync (optimized sync) fails upon first sync after capsule was installed.

Version-Release number of selected component (if applicable):
satellite-capsule-6.10.0-0.6.beta.el7sat.noarch

How reproducible:
Unsure

Steps to Reproduce:
1. Installed capsule 6.10
2. Launched a sync
3.

Actual results:
~~~
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(Server, Server, Server, variant, Packages, 00282cb5-53fb-4e49-b1db-ccf9ffdf7e17) already exists.
duplicate key value violates unique constraint "rpm_variant_variant_id_uid_name_type_764da894_uniq"
DETAIL:  Key (variant_id, uid, name, type, packages, distribution_tree_id)=(Server, Server, Server, variant, Packages, 7655b202-cabd-4740-ac09-0a892125e7e1) already exists.
deadlock detected
DETAIL:  Process 9122 waits for ShareLock on transaction 579218; blocked by process 12902.
Process 12902 waits for ShareLock on transaction 578810; blocked by process 9122.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (30507,3) in relation "rpm_package_pkgId_key"
~~~

Expected results:
no errors

Additional info:

Comment 2 Kenny Tordeurs 2021-08-25 15:00:24 UTC
production.log has the following traceback:

2021-08-25T15:46:53 [E|bac|f1534055] deadlock detected
 f1534055 | DETAIL:  Process 9122 waits for ShareLock on transaction 579218; blocked by process 12902.
 f1534055 | Process 12902 waits for ShareLock on transaction 578810; blocked by process 9122.
 f1534055 | HINT:  See server log for query details.
 f1534055 | CONTEXT:  while inserting index tuple (30507,3) in relation "rpm_package_pkgId_key"
 f1534055 |  (Katello::Errors::Pulp3Error)
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/polling.rb:22:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/cancellable.rb:14:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:572:in `block (3 levels) in execute_run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/middleware/record_smart_proxy_sync_history.rb:26:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:32:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/middleware/remote_action.rb:16:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/progress.rb:17:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.3/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.3/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.3/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/world.rb:31:in `execute'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:571:in `block (2 levels) in execute_run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:570:in `catch'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:570:in `block in execute_run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in `block in with_error_handling'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in `catch'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in `with_error_handling'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:565:in `execute_run'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:286:in `execute'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/director.rb:94:in `execute'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors.rb:18:in `run_user_code'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 f1534055 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 f1534055 | [ sidekiq ]
 f1534055 | [ concurrent-ruby ]

Comment 3 Brad Buckingham 2021-08-30 12:24:41 UTC
Bug 1974482 was opened and closed as not reproducible.  Based upon this one, it appears the issue does still exist.

Another related one may be bug 1996480.

Comment 4 pulp-infra@redhat.com 2021-09-10 18:08:59 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2021-09-10 18:09:00 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 6 Grant Gainey 2021-09-21 13:34:21 UTC
*** Bug 1996480 has been marked as a duplicate of this bug. ***

Comment 7 pulp-infra@redhat.com 2021-09-24 13:15:13 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 8 pulp-infra@redhat.com 2021-09-24 13:15:14 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 9 pulp-infra@redhat.com 2021-09-28 16:11:21 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 10 pulp-infra@redhat.com 2021-09-28 21:06:09 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 11 pulp-infra@redhat.com 2021-09-29 02:17:51 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 12 pulp-infra@redhat.com 2021-09-29 04:10:23 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 13 pulp-infra@redhat.com 2021-09-29 05:07:23 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 14 pulp-infra@redhat.com 2021-09-29 19:06:55 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 15 pulp-infra@redhat.com 2021-09-29 19:07:07 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 16 Danny Synk 2021-10-04 15:19:40 UTC
Verified on Satellite 6.10, snap 21.

Steps to Test:
1. Deploy a Satellite 6.10 and a Capsule 6.10.
2. Configure the Capsule with a default SSL certificate from the Satellite.
3. Create a new lifecycle environment on the Satellite.
4. Synchronize one repository to the Satellite.
5. Create a new content view, add the repository synced in step 4, publish the first version of the content view, and promote the version to the lifecycle environment created in step 3.
5. In the Satellite webUI, on the Infrastructure > Capsules > [Capsule name] > Edit > Lifecycle Environments tab, add the lifecycle environment created in step 3 to the Capsule.
6. On the Infrastructure > Capsules > [Capsule name] page, select "Optimized Sync" from the "Synchronize" dropdown menu.

Expected Results:
The Capsule synchronizes successfully with no errors.

Actual Results:
The Capsule synchronizes successfully with no error.

Comment 19 errata-xmlrpc 2021-11-16 14:13:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4702