New commit detected on ManageIQ/manageiq/hammer: https://github.com/ManageIQ/manageiq/commit/8da6f7b46b3c5d73082dbaa9389b787642b33597 commit 8da6f7b46b3c5d73082dbaa9389b787642b33597 Author: Satoe Imaishi <simaishi> AuthorDate: Mon Jan 6 15:56:31 2020 -0500 Commit: Satoe Imaishi <simaishi> CommitDate: Mon Jan 6 15:56:31 2020 -0500 Merge pull request #19638 from jrafanie/fix_frozen_hash_updating_worker [IVANCHUK] Query current/starting miq_workers, bypassing cache/stale objects (cherry picked from commit 06d8ef03af7f5fec914b7dc0933adb8e627d4dc7) Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1788289 app/models/miq_server/worker_management/heartbeat.rb | 17 +- app/models/mixins/miq_web_server_worker_mixin.rb | 1 + spec/lib/workers/heartbeat_spec.rb | 22 + 3 files changed, 32 insertions(+), 8 deletions(-)
Hi Joe, can you help me recreate this bug? What steps do I need to follow to reproduce this?
Ah, great question. Sorry I forgot to include this information. There is a specific situation you need to recreate: 1) Worker that consumes too much memory, exceeding the threshold 2) and will not finish it's task quickly 3) and therefore will not get the message to exit fast enough To do this, you need to get very lucky with a custom database OR you can fake it. I chose to fake it. a) get the manageiq-api/cfme-api location bundle show manageiq-api b) edit the events_controller in this location: vi /opt/rh/cfme-gemset/bundler/gems/cfme-api-258310204c75/app/controllers/api/events_controller.rb [root@localhost ~]# cat /opt/rh/cfme-gemset/bundler/gems/cfme-api-258310204c75/app/controllers/api/events_controller.rb module Api class EventsController < BaseController def index x = "a" * 1000000000 # Add a line to create lots of memory sleep(rand(60)) # Add a line to pretend to be busy for a random number of seconds super end end c) Start your server. d) Configure for only 1 web service worker e) Hit the api endpoint: curl -k https://admin:smartvm@localhost/api/events This should make the worker grow to 1 GB+ when you hit the events controller and because it will sleep for up to 60 seconds before returning from the index, you should get the server "can't modify frozen hash" error in the logs. You may have to do step e) multiple times.
Thanks Joe! I was able to recreate it and also verify it on 5.10.14. I don't see the error message in the logs. Marking the bug as Verified. Verified. Fixed on 5.10.14.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0189