Description of problem: Due to original dynflow scheduling, some tasks can be discriminated in favour of newer tasks when under heavy load: even though the overall execution takes overaly the same amount, there are higher spikes on longer-running tasks. Version-Release number of selected component (if applicable): How reproducible: requires special setup to observer reliably Steps to Reproduce: 1. trigger a lot (200+) of small execution plans (applicability on large amount hosts is a good example) 2. watch task duration of the tasks Actual results: some tasks finish relatively fast, while others wait unexpectedly long to finish: newer tasks of the same type get finished much sooner, than other older tasks. Expected results: FIFO: tasks triggered sooner should finish sooner, than newer tasks of the same type and complexity.
Created redmine issue https://projects.theforeman.org/issues/24435 from this bug
The fix for the bug is proposed upstream https://github.com/Dynflow/dynflow/pull/293
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/24435 has been resolved.
Did a quick test on througput of a Satellite 6.4 server with this modification applied # DEFAULT SETTINGS (1 executor) 4000 Host::Update & GenerateApplicability tasks 18 CV Publishes Duration: Started at: 2019-06-13 16:43:53 UTC Ended at: 2019-06-13 17:19:02 UTC DURATION: 36min # 2 EXECUTORS 4000 Host::Update & GenerateApplicability tasks 18 CV Publishes Duration: Started at: 2019-06-13 17:36:01 UTC Ended at: 2019-06-13 17:59:10 UTC DURATION: 23min # 2 EXECUTORS & DYNFLOW FIFO MODIFICATION 4000 Host::Update & GenerateApplicability tasks 18 CV Publishes Started at: 2019-06-13 18:12:23 UTC Ended at: 2019-06-13 18:25:31 UTC 4000 tasks 18 CV Publishes DURATION: 13min Almost a 100% throughput improvement
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:3172