Bug 1602110

Summary:	The number of MAX_RETRIES exceeded: PersistenceError in executor: terminating when running tasks are removed from database
Product:	Red Hat Satellite	Reporter:	Ivan Necas <inecas>
Component:	Tasks Plugin	Assignee:	Ivan Necas <inecas>
Status:	CLOSED ERRATA	QA Contact:	Jan Hutař <jhutar>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	Unspecified	CC:	andrew.schofield, aruzicka, cmarinea, inecas, jhutar, pcreech, pmoravec
Target Milestone:	6.5.0	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-05-14 12:37:37 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ivan Necas 2018-07-17 20:19:59 UTC

Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:
occasionally

Steps to Reproduce:
1. schedule remote execution job against larger number of hosts (300+) - to ensure some tasks will be actually running during the cleanup
2. foreman-rake foreman_tasks:cleanup TASK_SEARCH='' STATES='running'
3. schedule new remote execution job

Expected results:

There is no info about "PersistenceError in executor: terminating" in production.log, the executor is able to process new requests after this incident without restart.

Actual results:

There is the following error, followed by the executor either restarting or getting stuck.

2018-06-20 18:36:27 [foreman-tasks/dynflow] [E] The number of MAX_RETRIES exceeded
2018-06-20 18:36:27 [foreman-tasks/dynflow] [F] PersistenceError in executor: terminating
2018-06-20 18:36:27 [foreman-tasks/dynflow] [E] PG::Error: ERROR: insert or update on table "dynflow_actions" violates foreign key constraint "dynflow_actions_execution_plan_uuid_fkey"
| DETAIL: Key (execution_plan_uuid)=(6efc7c85-46ed-4119-9f35-710d12e4f72b) is not present in table "dynflow_execution_plans". 
| (Sequel::ForeignKeyConstraintViolation) 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:180:in `exec' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:180:in `block in execute_query' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/logging.rb:33:in `log_yield' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:180:in `execute_query' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:167:in `block in execute' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:143:in `check_disconnect_errors' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:167:in `execute' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:511:in `_execute' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:335:in `block (2 levels) in execute' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:532:in `check_database_errors' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:335:in `block in execute' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/connecting.rb:250:in `block in synchronize'

Also trying to rerun other jobs after this incided without restarting foreman-tasks might not work.

Additional info:

Comment 1 Ivan Necas 2018-07-17 20:22:31 UTC

Created redmine issue https://projects.theforeman.org/issues/24278 from this bug

Comment 3 Satellite Program 2018-07-25 14:09:47 UTC

Upstream bug assigned to inecas

Comment 4 Satellite Program 2018-07-25 14:09:50 UTC

Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/24278 has been resolved.

Comment 10 errata-xmlrpc 2019-05-14 12:37:37 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222