Bug 1849869 - Unable to recycle the dynflow executor
Summary: Unable to recycle the dynflow executor
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Tasks Plugin
Version: 6.7.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: 6.8.0
Assignee: Adam Ruzicka
QA Contact: Peter Ondrejka
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-23 04:19 UTC by Hao Chang Yu
Modified: 2023-12-15 18:14 UTC (History)
5 users (show)

Fixed In Version: tfm-rubygem-dynflow-1.4.6
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 13:03:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 30192 0 Normal New Unable to recycle the dynflow executor 2021-02-21 22:59:02 UTC
Github Dynflow dynflow pull 357 0 None closed Use correct method name for resolving events 2021-02-21 22:59:02 UTC
Red Hat Product Errata RHSA-2020:4366 0 None None None 2020-10-27 13:03:53 UTC

Description Hao Chang Yu 2020-06-23 04:19:56 UTC
Description of problem:
Unable to recycle executor when the memory limit is reached. The world got terminated but no new world is created. The executor process doesn't exit.

Steps to Reproduce:
Setting the "EXECUTOR_MEMORY_LIMIT" to something very low which will trigger the refresh pretty much 60 seconds after the starting the executor.

----------------------------
#Set memory limit for executor process, before it's restarted automatically
EXECUTOR_MEMORY_LIMIT=200mb

#Set delay before first memory polling to let executor initialize (in sec)
EXECUTOR_MEMORY_MONITOR_DELAY=60 #default: 2 hours
----------------------------


Observed that:
- the world got terminated when the memory limit is reached.
- memory is not freed and no more executor showing in the Dynflow status page.
- running task will enter paused stated and never able to resume/cancel even after restarting the dynflowd.

Log:
Everything ready for world: da8f1461-7891-47a1-a39a-2ff44c0492bd
World has been terminated
<getting dead letter error>

# ps -aux | grep -i dynflow
foreman  18794 22.8  2.0 2572916 409472 ?      Sl   17:37   1:05 dynflow_executor
foreman  18795  0.0  0.6 581248 126556 ?       Sl   17:37   0:00 dynflow_executor_monitor

Same behaviour on both Satellite 6.6 and 6.7 but working on Satellite 6.5. 


Actual results:
failed to recycle executor

Expected results:
executor can be recycled successfully

Comment 2 Bryan Kearney 2020-06-23 08:05:47 UTC
Upstream bug assigned to aruzicka

Comment 3 Peter Ondrejka 2020-07-14 11:48:50 UTC
Verified on Satellite 6.8 snap 8

Comment 6 errata-xmlrpc 2020-10-27 13:03:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.8 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4366


Note You need to log in before you can comment on or make changes to this bug.