Bug 1460263 - shutdown_and_exit messages get marked as error and never removed from miq_queue table
Summary: shutdown_and_exit messages get marked as error and never removed from miq_que...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance
Version: 5.8.0
Hardware: All
OS: All
medium
medium
Target Milestone: GA
: 5.10.0
Assignee: Yuri Rudman
QA Contact: Tasos Papaioannou
URL:
Whiteboard: appliance
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-06-09 14:06 UTC by Tasos Papaioannou
Modified: 2019-02-07 23:02 UTC (History)
6 users (show)

Fixed In Version: 5.10.0.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-02-07 23:02:34 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:0212 None None None 2019-02-07 23:02:39 UTC

Description Tasos Papaioannou 2017-06-09 14:06:42 UTC
Description of problem:

From appliance_console, select Stop EVM Server Processes, then Start EVM Server Processes. The processes stop and start as expected, but the shutdown_and_exit messages stick around in the database w/ a state of 'error':

vmdb_production=# select id, method_name, state, created_on, updated_on, queue_name, class_name, instance_id from miq_queue;
  id   |    method_name    | state |         created_on         |         updated_on         | queue_name | class_name | instance_id 
-------+-------------------+-------+----------------------------+----------------------------+------------+------------+-------------
 21775 | shutdown_and_exit | error | 2017-06-08 17:45:03.548581 | 2017-06-08 17:47:04.980664 | miq_server | MiqServer  |           2
 23188 | shutdown_and_exit | error | 2017-06-08 19:57:34.96574  | 2017-06-08 19:58:49.018568 | miq_server | MiqServer  |           2
 23094 | shutdown_and_exit | error | 2017-06-08 19:51:55.54475  | 2017-06-08 19:53:23.717701 | miq_server | MiqServer  |           2
 22578 | shutdown_and_exit | error | 2017-06-08 19:01:42.738824 | 2017-06-08 19:04:35.980748 | miq_server | MiqServer  |           2
 21704 | shutdown_and_exit | error | 2017-06-08 17:41:32.127746 | 2017-06-08 17:43:04.36108  | miq_server | MiqServer  |           2
 21587 | shutdown_and_exit | error | 2017-06-08 17:25:42.082366 | 2017-06-08 17:35:38.89939  | miq_server | MiqServer  |           2
(6 rows)

Version-Release number of selected component (if applicable):

5.8.0.17

How reproducible:

100%

Steps to Reproduce:
1.) Stop then start EVM processes from appliance_console.
2.) View miq_queue entries in psql or rails console.
3.) See all historical shutdown_and_exit messages, marked as 'error'.

Actual results:

shutdown_and_exit messages not cleaned out of db after startup.

Expected results:

shutdown_and_exit messages cleaned out of db after startup.

Additional info:

In /var/www/miq/vmdb/app/models/miq_queue.rb, the atStartup method finds all queue messages with a state of 'dequeue' handled by the server, and updates their state to 'error'.

****
  def self.atStartup
    _log.info("Cleaning up queue messages...")
    MiqQueue.where(:state => STATE_DEQUEUE).each do |message|
      if message.handler.nil?
        _log.warn("Cleaning message in dequeue state without worker: #{format_full_log_msg(message)}")
      else
        handler_server = message.handler            if message.handler.kind_of?(MiqServer)
        handler_server = message.handler.miq_server if message.handler.kind_of?(MiqWorker)
        next unless handler_server == MiqServer.my_server
        _log.warn("Cleaning message: #{format_full_log_msg(message)}")
      end
      message.update_attributes(:state => STATE_ERROR) rescue nil
    end
    _log.info("Cleaning up queue messages... Complete")
  end
****

It's not clear that the shutdown_and_exit msg should be marked as 'error', since it did succeed. But either way, nothing cleans up the msg after it's marked as 'error', such as by calling the delivered method.

Comment 3 CFME Bot 2018-05-03 15:31:34 UTC
New commit detected on ManageIQ/manageiq/master:

https://github.com/ManageIQ/manageiq/commit/87cbb0cc69bf7cfad3c96ace92ae41e6f699f290
commit 87cbb0cc69bf7cfad3c96ace92ae41e6f699f290
Author:     Yuri Rudman <yrudman@redhat.com>
AuthorDate: Tue May  1 14:12:18 2018 -0400
Commit:     Yuri Rudman <yrudman@redhat.com>
CommitDate: Tue May  1 14:12:18 2018 -0400

    remove shutdown_and_exit message from queue on server start if message dequeued#
    Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1460263

 app/models/miq_server/at_startup.rb | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

Comment 4 CFME Bot 2018-05-03 15:33:55 UTC
New commit detected on ManageIQ/manageiq/master:

https://github.com/ManageIQ/manageiq/commit/87cbb0cc69bf7cfad3c96ace92ae41e6f699f290
commit 87cbb0cc69bf7cfad3c96ace92ae41e6f699f290
Author:     Yuri Rudman <yrudman@redhat.com>
AuthorDate: Tue May  1 14:12:18 2018 -0400
Commit:     Yuri Rudman <yrudman@redhat.com>
CommitDate: Tue May  1 14:12:18 2018 -0400

    remove shutdown_and_exit message from queue on server start if message dequeued#
    Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1460263

 app/models/miq_server/at_startup.rb | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

Comment 5 Tasos Papaioannou 2018-06-26 16:11:35 UTC
Verified on 5.10.0.1.

Comment 6 errata-xmlrpc 2019-02-07 23:02:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:0212


Note You need to log in before you can comment on or make changes to this bug.