Bug 1654347 - Satellite may create duplicate CreateRssNotifications tasks after restarting foreman tasks
Summary: Satellite may create duplicate CreateRssNotifications tasks after restarting ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Tasks Plugin
Version: 6.4
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: 6.8.0
Assignee: Adam Ruzicka
QA Contact: Peter Ondrejka
URL:
Whiteboard:
Depends On:
Blocks: 1122832
TreeView+ depends on / blocked
 
Reported: 2018-11-28 15:00 UTC by Hao Chang Yu
Modified: 2020-10-27 12:58 UTC (History)
11 users (show)

Fixed In Version: foreman-2.1.0-0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1851130 (view as bug list)
Environment:
Last Closed: 2020-10-27 12:58:03 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 25723 0 Normal Closed Foreman may create duplicate CreateRssNotifications tasks after restarting foreman tasks 2021-01-06 15:43:22 UTC
Red Hat Product Errata RHSA-2020:4366 0 None None None 2020-10-27 12:58:21 UTC

Description Hao Chang Yu 2018-11-28 15:00:01 UTC
Description of problem:
Satellite may create duplicate CreateRssNotifications, CreatePulpDiskSpaceNotifications or SendExpireSoonNotifications when restart foreman-tasks.

# su - postgres -c "psql -d foreman -c 'select label,count(label),state from foreman_tasks_tasks where state <> '\''stopped'\'' group by label,state;'"
                 label                  | count |   state   
----------------------------------------+-------+-----------
 CreateRssNotifications                 |     2 | scheduled
 CreatePulpDiskSpaceNotifications       |     1 | scheduled
 SendExpireSoonNotifications            |     1 | scheduled


The "spawn_if_missing" function is looking for task that is in "scheduled" state during the dynflow startup. If no "CreateRssNotifications" is in "schdeduled" state, then create one. I suspect that the task was not in "scheduled" state previously before Dynflow got terminated and cause Dynflow to create a duplicate task during startup.

/usr/share/foreman/app/jobs/application_job.rb

  def self.spawn_if_missing(world)
    return if (Foreman.in_rake? && !Foreman.in_rake?('dynflow:executor')) || Rails.env.test?

    pending_jobs = world.persistence.find_execution_plans(filters: { :state => 'scheduled' })
    scheduled_job = pending_jobs.select do |job|
      delayed_plan = world.persistence.load_delayed_plan job.id
      next unless delayed_plan.present?
      arg = delayed_plan.to_hash[:serialized_args].first
      arg.is_a?(Hash) && arg['job_class'] == self.to_s
    end

    # Schedule the job only if it doesn't exit yet
    self.perform_later if scheduled_job.blank?
  end

Comment 3 Adam Ruzicka 2018-11-29 12:09:08 UTC
I think I can see how this might happen, but I need to know many executors they have. 

Running "grep EXECUTORS_COUNT /etc/sysconfig/{dynflowd,foreman-tasks}" should show that.

Comment 5 Adam Ruzicka 2018-12-18 12:04:32 UTC
Created redmine issue http://projects.theforeman.org/issues/25723 from this bug

Comment 6 Bryan Kearney 2020-05-06 13:44:26 UTC
Upstream bug assigned to aruzicka@redhat.com

Comment 7 Bryan Kearney 2020-05-06 13:44:29 UTC
Upstream bug assigned to aruzicka@redhat.com

Comment 8 Bryan Kearney 2020-05-07 12:53:34 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/25723 has been resolved.

Comment 9 Peter Ondrejka 2020-05-26 12:26:52 UTC
Verified on Satellite 6.8 snap 1, on fresh machine after several restarts of dynflow-sidekiq@ services the count of the aforementioned tasks remains one

Comment 12 errata-xmlrpc 2020-10-27 12:58:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.8 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4366


Note You need to log in before you can comment on or make changes to this bug.