Created attachment 1217120 [details] current settings.yml from beta2 Description of problem: schedule worker, generic worker, priority worker and metrics_processor workers are not specified with sufficient memory limits (aka thresholds) to prevent frequent (eg, less than 300 seconds) worker restarts. Version-Release number of selected component (if applicable):5.7.0.x How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: support is encountering issues where all upgrades from 4.0 to 4.1 are encountering problems with the memory thresholds of these workers as being too small. the schedule worker is particularly troublesome because when it fails it puts at risk all of the schedules that are about to be queued for execution and can prevent all sorts of scheduled activities from actually being queued for exection (eg, reports, widgets, schedule activities, etc) I am attaching the settings.yml file from 5.7.0.5 beta2 as a representation of current default settings that are in need of being increased.
Can GSS provide recommendations for the default threshold for each of the worker types that are currently too low? What values were used in order to resolve the attached issues with real world providers?
the schedule worker threshold is recommended to be increased to 500MB. the metrics processor worker and priority workers is recommended to be increased to 600MB and the generic worker memory is recommended to be increased to 500MB.
Thanks Tom, created a PR to increase the thresholds: https://github.com/ManageIQ/manageiq/pull/12484
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/047f200dbe194432a84748d85cc59e1fe241cb42 commit 047f200dbe194432a84748d85cc59e1fe241cb42 Author: Joe Rafaniello <jrafanie> AuthorDate: Mon Nov 7 15:49:18 2016 -0500 Commit: Joe Rafaniello <jrafanie> CommitDate: Mon Nov 7 16:03:40 2016 -0500 Increase worker memory thresholds to avoid frequent restarts. https://bugzilla.redhat.com/show_bug.cgi?id=1391687 The prior values had not been increased to account for newer ruby versions that utilize a generational garbage collector which consumes more memory. Change default worker from 200 to 400 MB. ems_refresh_core_worker inherits this 400 MB default. Change default queue worker from 400 to 500 MB. generic_worker inherits this 500 MB value. Change ems_metrics_processor_worker from 400 to 600 MB. Change priority_worker from the old inherited queue worker value of 400 to a customized 600 MB. Change schedule_worker from 300 to 500 MB. config/settings.yml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
New commit detected on ManageIQ/manageiq/euwe: https://github.com/ManageIQ/manageiq/commit/a06b8a9355195d52875c4977c08c95d9756d6800 commit a06b8a9355195d52875c4977c08c95d9756d6800 Author: Gregg Tanzillo <gtanzill> AuthorDate: Mon Nov 7 18:07:00 2016 -0500 Commit: Oleg Barenboim <chessbyte> CommitDate: Mon Nov 7 18:15:58 2016 -0500 Merge pull request #12484 from jrafanie/increase_worker_memory_thresholds Increase worker memory thresholds to avoid frequent restarts. (cherry picked from commit d6f44669a4fb5de3a985b010b500763d375acf5a) https://bugzilla.redhat.com/show_bug.cgi?id=1391687 config/settings.yml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
Verified fixed in 5.8.0.12-rc1.20170425180304_4f35996.