Bug 1460263 - shutdown_and_exit messages get marked as error and never removed from miq_queue table
shutdown_and_exit messages get marked as error and never removed from miq_que...
Status: NEW
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance (Show other bugs)
5.8.0
All All
medium Severity medium
: GA
: cfme-future
Assigned To: Gregg Tanzillo
Tasos Papaioannou
appliance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-09 10:06 EDT by Tasos Papaioannou
Modified: 2017-09-21 13:45 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Tasos Papaioannou 2017-06-09 10:06:42 EDT
Description of problem:

From appliance_console, select Stop EVM Server Processes, then Start EVM Server Processes. The processes stop and start as expected, but the shutdown_and_exit messages stick around in the database w/ a state of 'error':

vmdb_production=# select id, method_name, state, created_on, updated_on, queue_name, class_name, instance_id from miq_queue;
  id   |    method_name    | state |         created_on         |         updated_on         | queue_name | class_name | instance_id 
-------+-------------------+-------+----------------------------+----------------------------+------------+------------+-------------
 21775 | shutdown_and_exit | error | 2017-06-08 17:45:03.548581 | 2017-06-08 17:47:04.980664 | miq_server | MiqServer  |           2
 23188 | shutdown_and_exit | error | 2017-06-08 19:57:34.96574  | 2017-06-08 19:58:49.018568 | miq_server | MiqServer  |           2
 23094 | shutdown_and_exit | error | 2017-06-08 19:51:55.54475  | 2017-06-08 19:53:23.717701 | miq_server | MiqServer  |           2
 22578 | shutdown_and_exit | error | 2017-06-08 19:01:42.738824 | 2017-06-08 19:04:35.980748 | miq_server | MiqServer  |           2
 21704 | shutdown_and_exit | error | 2017-06-08 17:41:32.127746 | 2017-06-08 17:43:04.36108  | miq_server | MiqServer  |           2
 21587 | shutdown_and_exit | error | 2017-06-08 17:25:42.082366 | 2017-06-08 17:35:38.89939  | miq_server | MiqServer  |           2
(6 rows)

Version-Release number of selected component (if applicable):

5.8.0.17

How reproducible:

100%

Steps to Reproduce:
1.) Stop then start EVM processes from appliance_console.
2.) View miq_queue entries in psql or rails console.
3.) See all historical shutdown_and_exit messages, marked as 'error'.

Actual results:

shutdown_and_exit messages not cleaned out of db after startup.

Expected results:

shutdown_and_exit messages cleaned out of db after startup.

Additional info:

In /var/www/miq/vmdb/app/models/miq_queue.rb, the atStartup method finds all queue messages with a state of 'dequeue' handled by the server, and updates their state to 'error'.

****
  def self.atStartup
    _log.info("Cleaning up queue messages...")
    MiqQueue.where(:state => STATE_DEQUEUE).each do |message|
      if message.handler.nil?
        _log.warn("Cleaning message in dequeue state without worker: #{format_full_log_msg(message)}")
      else
        handler_server = message.handler            if message.handler.kind_of?(MiqServer)
        handler_server = message.handler.miq_server if message.handler.kind_of?(MiqWorker)
        next unless handler_server == MiqServer.my_server
        _log.warn("Cleaning message: #{format_full_log_msg(message)}")
      end
      message.update_attributes(:state => STATE_ERROR) rescue nil
    end
    _log.info("Cleaning up queue messages... Complete")
  end
****

It's not clear that the shutdown_and_exit msg should be marked as 'error', since it did succeed. But either way, nothing cleans up the msg after it's marked as 'error', such as by calling the delivered method.

Note You need to log in before you can comment on or make changes to this bug.