Description of problem: Import task goes to a paused state and return error complaining about a deadlock when indexing repositories Version-Release number of selected component (if applicable): 6.12 How reproducible: Eventually Steps to Reproduce: Still unsure. Managed to reproduce once, but couldn't repeat it regularly. Saved logs from the time it happened on the lab. Actual results: import task get paused due to deadlock. Resuming the task works fine and it can finish. Expected results: no deadlock Additional info: ~~~ # hammer content-import library --path /var/lib/pulp/imports/Export-Library/1.0/2023-04-05T09-11-07-04-00/ --organization-id 1 [...................................................................... ] [31%] Error: PG::TRDeadlockDetected: ERROR: deadlock detected DETAIL: Process 2151861 waits for ShareLock on transaction 12856382; blocked by process 2151215. Process 2151215 waits for ShareLock on transaction 12856380; blocked by process 2151861. HINT: See server log for query details. CONTEXT: while inserting index tuple (1,1) in relation "katello_erratum_bugzillas" ~~~ DB log: ~~~ 2023-04-05 10:59:42 EDT ERROR: deadlock detected 2023-04-05 10:59:42 EDT DETAIL: Process 2151861 waits for ShareLock on transaction 12856382; blocked by process 2151215. Process 2151215 waits for ShareLock on transaction 12856380; blocked by process 2151861. Process 2151861: INSERT INTO "katello_erratum_bugzillas" ("bug_id","href","erratum_id") VALUES ('2131770', 'https://bugzilla.redhat.com/show_bug.cgi?id=2131770', 72684), ('2049850', 'https://bugzilla.redhat.com/show_bug.cgi?id=2049850', 72685), ('1766653', 'https://bugzilla.redhat.com/show_bug.cgi?id=1766653', 72685), ('2124088', 'https://bugzilla.redhat.com/show_bug.cgi?id=2124088', 72686), ('2116280', 'https://bugzilla.redhat.com/show_bug.cgi?id=2116280', 72686) ON CONFLICT ("erratum_id","bug_id","href") DO NOTHING RETURNING "id" Process 2151215: INSERT INTO "katello_erratum_bugzillas" ("bug_id","href","erratum_id") VALUES ('1766653', 'https://bugzilla.redhat.com/show_bug.cgi?id=1766653', 72685), ('2049850', 'https://bugzilla.redhat.com/show_bug.cgi?id=2049850', 72685), ('2131770', 'https://bugzilla.redhat.com/show_bug.cgi?id=2131770', 72684), ('2124088', 'https://bugzilla.redhat.com/show_bug.cgi?id=2124088', 72686), ('2116280', 'https://bugzilla.redhat.com/show_bug.cgi?id=2116280', 72686) ON CONFLICT ("erratum_id","bug_id","href") DO NOTHING RETURNING "id" 2023-04-05 10:59:42 EDT HINT: See server log for query details. 2023-04-05 10:59:42 EDT CONTEXT: while inserting index tuple (1,1) in relation "katello_erratum_bugzillas" 2023-04-05 10:59:42 EDT STATEMENT: INSERT INTO "katello_erratum_bugzillas" ("bug_id","href","erratum_id") VALUES ('2131770', 'https://bugzilla.redhat.com/show_bug.cgi?id=2131770', 72684), ('2049850', 'https://bugzilla.redhat.com/show_bug.cgi?id=2049850', 72685), ('1766653', 'https://bugzilla.redhat.com/show_bug.cgi?id=1766653', 72685), ('2124088', 'https://bugzilla.redhat.com/show_bug.cgi?id=2124088', 72686), ('2116280', 'https://bugzilla.redhat.com/show_bug.cgi?id=2116280', 72686) ON CONFLICT ("erratum_id","bug_id","href") DO NOTHING RETURNING "id" 2023-04-05 10:59:42 EDT LOG: duration: 1008.287 ms execute <unnamed>: INSERT INTO "katello_erratum_bugzillas" ("bug_id","href","erratum_id") VALUES ('1766653', 'https://bugzilla.redhat.com/show_bug.cgi?id=1766653', 72685), ('2049850', 'https://bugzilla.redhat.com/show_bug.cgi?id=2049850', 72685), ('2131770', 'https://bugzilla.redhat.com/show_bug.cgi?id=2131770', 72684), ('2124088', 'https://bugzilla.redhat.com/show_bug.cgi?id=2124088', 72686), ('2116280', 'https://bugzilla.redhat.com/show_bug.cgi?id=2116280', 72686) ON CONFLICT ("erratum_id","bug_id","href") DO NOTHING RETURNING "id" ~~~ production.log: ~~~ 2023-04-05T10:59:42 [I|app|10bb9b2a] CONTENT_INDEX type=Katello::Erratum success=false duration=1166.52 2023-04-05T10:59:42 [E|bac|10bb9b2a] PG::TRDeadlockDetected: ERROR: deadlock detected 10bb9b2a | DETAIL: Process 2151861 waits for ShareLock on transaction 12856382; blocked by process 2151215. 10bb9b2a | Process 2151215 waits for ShareLock on transaction 12856380; blocked by process 2151861. 10bb9b2a | HINT: See server log for query details. 10bb9b2a | CONTEXT: while inserting index tuple (1,1) in relation "katello_erratum_bugzillas" 10bb9b2a | (ActiveRecord::Deadlocked) 10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `exec_params' 10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache' 10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads' 10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares' 10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads' 10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:674:in `block in exec_no_cache' 10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract_adapter.rb:728:in `block (2 levels) in log' 10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize' 10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt' 10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize' 10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt' 10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize' 10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract_adapter.rb:727:in `block in log' 10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument' 10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract_adapter.rb:718:in `log' 10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:673:in `exec_no_cache' 10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:654:in `execute_and_clear' 10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/postgresql/database_statements.rb:98:in `exec_query' 10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract/database_statements.rb:153:in `exec_insert_all' 10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `exec_insert_all' 10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/insert_all.rb:27:in `execute' 10bb9b2a | /usr/share/gems/gems/activerecord-6.0.6/lib/active_record/persistence.rb:124:in `insert_all' 10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/pulp3/erratum.rb:60:in `insert_child_associations' 10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/content_unit_indexer.rb:60:in `import_associations' 10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/content_unit_indexer.rb:50:in `block in import_all' 10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/pulp3/pulp_content_unit.rb:97:in `yield' 10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/pulp3/pulp_content_unit.rb:97:in `block (2 levels) in pulp_units_batch_for_repo' 10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/pulp3/pulp_content_unit.rb:85:in `loop' 10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/pulp3/pulp_content_unit.rb:85:in `block in pulp_units_batch_for_repo' 10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/content_unit_indexer.rb:26:in `each' 10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/services/katello/content_unit_indexer.rb:26:in `import_all' 10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/models/katello/repository.rb:926:in `block (2 levels) in index_content' 10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/lib/katello/logging.rb:8:in `time' 10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/models/katello/repository.rb:925:in `block in index_content' 10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/models/katello/repository.rb:924:in `each' 10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/models/katello/repository.rb:924:in `index_content' 10bb9b2a | /usr/share/gems/gems/katello-4.5.0.24/app/lib/actions/katello/repository/index_content.rb:22:in `run' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass' 10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run' 10bb9b2a | /usr/share/gems/gems/activesupport-6.0.6/lib/active_support/execution_wrapper.rb:91:in `wrap' 10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:31:in `with_progress_calculation' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:17:in `run' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass' 10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/load_setting_values.rb:20:in `run' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass' 10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run' 10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id' 10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass' 10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run' 10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone' 10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass' 10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run' 10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies' 10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in `run' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass' 10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run' 10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user' 10bb9b2a | /usr/share/gems/gems/foreman-tasks-6.0.3/app/lib/actions/middleware/keep_current_user.rb:15:in `run' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/world.rb:31:in `execute' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `catch' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `block in execute_run' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `block in with_error_handling' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `catch' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `with_error_handling' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:575:in `execute_run' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:296:in `execute' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/director.rb:69:in `execute' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors.rb:18:in `run_user_code' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform' 10bb9b2a | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform' 10bb9b2a | [ sidekiq ] 10bb9b2a | [ concurrent-ruby ] ~~~