Bug 1788289

New commit detected on ManageIQ/manageiq/hammer:

https://github.com/ManageIQ/manageiq/commit/8da6f7b46b3c5d73082dbaa9389b787642b33597
commit 8da6f7b46b3c5d73082dbaa9389b787642b33597
Author:     Satoe Imaishi <simaishi>
AuthorDate: Mon Jan  6 15:56:31 2020 -0500
Commit:     Satoe Imaishi <simaishi>
CommitDate: Mon Jan  6 15:56:31 2020 -0500

    Merge pull request #19638 from jrafanie/fix_frozen_hash_updating_worker

    [IVANCHUK] Query current/starting miq_workers, bypassing cache/stale objects

    (cherry picked from commit 06d8ef03af7f5fec914b7dc0933adb8e627d4dc7)

    Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1788289

 app/models/miq_server/worker_management/heartbeat.rb | 17 +-
 app/models/mixins/miq_web_server_worker_mixin.rb | 1 +
 spec/lib/workers/heartbeat_spec.rb | 22 +
 3 files changed, 32 insertions(+), 8 deletions(-)

Hi Joe, can you help me recreate this bug? What steps do I need to follow to reproduce this?

Ah, great question.  Sorry I forgot to include this information.

There is a specific situation you need to recreate:
1) Worker that consumes too much memory, exceeding the threshold
2) and will not finish it's task quickly
3) and therefore will not get the message to exit fast enough

To do this, you need to get very lucky with a custom database OR you can fake it.

I chose to fake it.

a) get the manageiq-api/cfme-api location
bundle show manageiq-api

b) edit the events_controller in this location:
vi /opt/rh/cfme-gemset/bundler/gems/cfme-api-258310204c75/app/controllers/api/events_controller.rb

[root@localhost ~]# cat /opt/rh/cfme-gemset/bundler/gems/cfme-api-258310204c75/app/controllers/api/events_controller.rb

module Api
  class EventsController < BaseController
    def index
      x = "a" * 1000000000  # Add a line to create lots of memory
      sleep(rand(60))       # Add a line to pretend to be busy for a random number of seconds
      super
    end
  end

c) Start your server.  
d) Configure for only 1 web service worker
e) Hit the api endpoint:
curl -k https://admin:smartvm@localhost/api/events

This should make the worker grow to 1 GB+ when you hit the events controller
and because it will sleep for up to 60 seconds before returning from the index,
you should get the server "can't modify frozen hash" error in the logs.

You may have to do step e) multiple times.

Thanks Joe!
I was able to recreate it and also verify it on 5.10.14. I don't see the error message in the logs. Marking the bug as Verified.

Verified. Fixed on 5.10.14.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0189