Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1585275

Summary: Dynflow executors invalid and dynflow workers periodically go invalid
Product: Red Hat Satellite Reporter: Dylan Gross <dgross>
Component: Tasks PluginAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED WONTFIX QA Contact: Peter Ondrejka <pondrejk>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.3.1CC: aruzicka, cduryee, inecas, jdickers
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-15 20:30:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dylan Gross 2018-06-01 17:18:00 UTC
Description of problem:

  Dynflow executors invalid and dynflow workers go invalid periodically when checked via the Dynflow RFE:  https://satellite.example.com/foreman_tasks/dynflow/worlds/

Version-Release number of selected component (if applicable):

  Red Hat Satellite 6.3.1   (Plus Hotfix tfm-rubygem-katello_ostree-3.4.5.64-2.HOTFIXRHBZ1560740_1563002_1567978.el7sat)

How reproducible:  Unknown, but recurring

Actual results:

   Dynflow executors and workers go invalid.    They must be manually invalidated, which will restart the processes and the queue can begin processing again.   If it's not manually invalidated, the queue will back up and many tasks within Satellite do not complete.

Expected results:

   The executors and workers remain running.

Comment 3 Ivan Necas 2018-06-04 13:48:38 UTC
I see a lot of, that indicate, that there has been a deletion of tasks going on that were actually running: I can imagine this leading to the dynflow executor to became unresponsive, but need to try out reproducing with my lab setup to see if that's the right clue.

E, [2018-06-01T18:30:24.730575 #25020] ERROR -- /parallel-executor-core/pool/worker-2: searching: action by: {:execution_plan_uuid=>"234bec32-7a05-4309-9000-a572977ba262", :id=>8247} (KeyError)
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/persistence_adapters/sequel.rb:278:in `load_record'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/persistence_adapters/sequel.rb:146:in `load_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/persistence.rb:21:in `load_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:21:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/director.rb:55:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/executors/parallel/worker.rb:11:in `on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/context.rb:46:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'

Comment 10 Ivan Necas 2018-06-13 19:46:17 UTC
*** Bug 1588779 has been marked as a duplicate of this bug. ***

Comment 11 Ivan Necas 2018-06-13 19:49:38 UTC
FYI: there is another bug tracking the issue with the status check for foreman-tasks not working reliably when under heavier load https://bugzilla.redhat.com/show_bug.cgi?id=1538688, might be considered as duplicate once we're sure that it's the case with this BZ as well.

Comment 19 Bryan Kearney 2019-12-03 16:34:54 UTC
The Satellite Team is attempting to provide an accurate backlog of bugzilla requests which we feel will be resolved in the next few releases. We do not believe this bugzilla will meet that criteria, and have plans to close it out in 1 month. This is not a reflection on the validity of the request, but a reflection of the many priorities for the product. If you have any concerns about this, feel free to contact Red Hat Technical Support or your account team. If we do not hear from you, we will close this bug out. Thank you.

Comment 20 Bryan Kearney 2020-01-15 20:30:42 UTC
Thank you for your interest in Satellite 6. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this, please do not reopen. Instead, feel free to contact Red Hat Technical Support. Thank you.