Bug 1602110 - The number of MAX_RETRIES exceeded: PersistenceError in executor: terminating when running tasks are removed from database
Summary: The number of MAX_RETRIES exceeded: PersistenceError in executor: terminating...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Tasks Plugin
Version: Unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
medium vote
Target Milestone: Released
Assignee: Ivan Necas
QA Contact: Jan Hutař
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-17 20:19 UTC by Ivan Necas
Modified: 2019-10-07 17:17 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-14 12:37:37 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:1222 None None None 2019-05-14 12:37:44 UTC
Foreman Issue Tracker 24278 None None None 2018-07-17 20:22:33 UTC

Description Ivan Necas 2018-07-17 20:19:59 UTC
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:
occasionally

Steps to Reproduce:
1. schedule remote execution job against larger number of hosts (300+) - to ensure some tasks will be actually running during the cleanup
2. foreman-rake foreman_tasks:cleanup TASK_SEARCH='' STATES='running'
3. schedule new remote execution job

Expected results:

There is no info about "PersistenceError in executor: terminating" in production.log, the executor is able to process new requests after this incident without restart.

Actual results:

There is the following error, followed by the executor either restarting or getting stuck.

2018-06-20 18:36:27 [foreman-tasks/dynflow] [E] The number of MAX_RETRIES exceeded
2018-06-20 18:36:27 [foreman-tasks/dynflow] [F] PersistenceError in executor: terminating
2018-06-20 18:36:27 [foreman-tasks/dynflow] [E] PG::Error: ERROR: insert or update on table "dynflow_actions" violates foreign key constraint "dynflow_actions_execution_plan_uuid_fkey"
| DETAIL: Key (execution_plan_uuid)=(6efc7c85-46ed-4119-9f35-710d12e4f72b) is not present in table "dynflow_execution_plans". 
| (Sequel::ForeignKeyConstraintViolation) 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:180:in `exec' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:180:in `block in execute_query' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/logging.rb:33:in `log_yield' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:180:in `execute_query' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:167:in `block in execute' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:143:in `check_disconnect_errors' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:167:in `execute' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:511:in `_execute' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:335:in `block (2 levels) in execute' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:532:in `check_database_errors' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:335:in `block in execute' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/connecting.rb:250:in `block in synchronize'

Also trying to rerun other jobs after this incided without restarting foreman-tasks might not work.

Additional info:

Comment 1 Ivan Necas 2018-07-17 20:22:31 UTC
Created redmine issue https://projects.theforeman.org/issues/24278 from this bug

Comment 3 pm-sat@redhat.com 2018-07-25 14:09:47 UTC
Upstream bug assigned to inecas@redhat.com

Comment 4 pm-sat@redhat.com 2018-07-25 14:09:50 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/24278 has been resolved.

Comment 10 errata-xmlrpc 2019-05-14 12:37:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222


Note You need to log in before you can comment on or make changes to this bug.