Bug 1687771

Description of problem:
There are scenarios (based on a race condition or maybe call flow used for triggering the task?) where a task is planning for a while, and if dynflowd is restarted that time, the task sits "planning" forever.

Particular example (very visible without fix for bz1673447): see reproducer steps.

It is assumed https://github.com/Dynflow/dynflow/pull/303 fixes this.


Version-Release number of selected component (if applicable):
6.4.2 (or anything older)


How reproducible:
very likely (scale the test more to have better chance)


Steps to Reproduce:
1. Have more repos, more LEs and few Capsules

2. Create and publish many CVs with even identical content (one or two small repos, e.g.)

3. Promote many CVs to next LE, e.g. via:

for i in $(seq 1 20); do 
  hammer content-view version promote --content-view CV_${i} --organization-id 1 --from-lifecycle-environment-id 1 --to-lifecycle-environment-id 2 --async &
  sleep 1
done

4. monitor tasks status summary e.g. via:

sudo su - postgres -c "psql -d foreman -c 'select label,count(label),state,result from foreman_tasks_tasks where state <> '\''stopped'\'' group by label,state,result ORDER BY label;'"

5. Once there will be more Actions::Katello::CapsuleContent::Sync tasks in planning state, restart dynflowd:

service dynflowd restart

6. Monitor the tasks status summary until all Capsule Sync tasks terminate


Actual results:
6. is waiting for Godot


Expected results:
6. all Sync tasks successfully complete after a reasonable time


Additional info: