Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1817728 - Default task polling is too frequent at scale
Summary: Default task polling is too frequent at scale
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Tasks Plugin
Version: 6.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: 6.8.0
Assignee: Adam Ruzicka
QA Contact: Peter Ondrejka
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-26 19:58 UTC by Mike McCune
Modified: 2023-12-15 17:34 UTC (History)
8 users (show)

Fixed In Version: tfm-rubygem-foreman-tasks-1.1.1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1832587 (view as bug list)
Environment:
Last Closed: 2020-10-27 13:01:14 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
tfm-rubygem-foreman-tasks hotfix RPM for Satellite 6.7.0 (1.88 MB, application/x-rpm)
2020-04-17 18:40 UTC, wclark 		no flags Details
UPDATED tfm-rubygem-foreman-tasks hotfix RPM (1.88 MB, application/x-rpm)
2020-04-28 21:09 UTC, wclark 		no flags Details
UPDATED hotfix RPM (1.88 MB, application/x-rpm)
2020-06-01 16:27 UTC, wclark 		no flags Details
FURTHER UPDATED Hotfix RPM (1.88 MB, application/x-rpm)
2020-06-03 16:57 UTC, wclark 		no flags Details
Show Obsolete (3) View All


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 29423 0 Normal Closed Default task polling is too frequent at scale 2021-02-04 23:26:01 UTC
Red Hat Knowledge Base (Solution) 5234901 0 None None None 2020-07-20 10:02:07 UTC
Red Hat Product Errata RHSA-2020:4366 0 None None None 2020-10-27 13:01:57 UTC

Description Mike McCune 2020-03-26 19:58:30 UTC 
The default polling back-off algorithm can still cause an overload of polling during long running tasks:


  def poll_intervals
    [0.5, 1, 2, 4, 8, 16] 
  end 

when there are large numbers of tasks, you can get 'storms' of status checks during long running tasks.

Updated and expanded array has shown improvement in working conditions when allowing longer tasks a greater time period of checking in:

    [0.5, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024] 

We are going to update the default array to the above value and offer a configuration setting so it can be tuned to specific environments.
Comment 3 Adam Ruzicka 2020-03-27 11:22:19 UTC 
Created redmine issue https://projects.theforeman.org/issues/29423 from this bug
Comment 4 Bryan Kearney 2020-03-27 16:02:28 UTC 
Upstream bug assigned to aruzicka
Comment 5 Bryan Kearney 2020-03-27 16:02:30 UTC 
Upstream bug assigned to aruzicka
Comment 6 wclark 2020-04-17 18:40:18 UTC 
Created attachment 1679728 [details]
tfm-rubygem-foreman-tasks hotfix RPM for Satellite 6.7.0
Comment 7 wclark 2020-04-17 18:44:49 UTC 
HOTFIX is available for Satellite 6.7.0

INSTALLATION INSTRUCTIONS:

1. Make a complete backup or snapshot of Satellite server

2. Download the attached file tfm-rubygem-foreman-tasks-0.17.5.2-2.HOTFIXRHBZ1817728.el7sat.noarch.rpm and copy it to Satellite server

3. # yum install ./tfm-rubygem-foreman-tasks-0.17.5.2-2.HOTFIXRHBZ1817728.el7sat.noarch.rpm --disableplugin=foreman-protector

4. # systemctl restart dynflowd httpd
Comment 10 wclark 2020-04-28 21:09:43 UTC 
Created attachment 1682631 [details]
UPDATED tfm-rubygem-foreman-tasks hotfix RPM

This also includes the hotfix for https://bugzilla.redhat.com/show_bug.cgi?id=1824931

The filename has changed but installation instructions are otherwise the same as before
Comment 12 Bryan Kearney 2020-05-06 13:45:58 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/29423 has been resolved.
Comment 13 wclark 2020-06-01 16:27:54 UTC 
Created attachment 1694147 [details]
UPDATED hotfix RPM

INSTALLATION INSTRUCTIONS:

1. Make a complete backup or snapshot of Satellite server

2. Download the attached file tfm-rubygem-foreman-tasks-0.17.5.2-4.HOTFIXRHBZ1817728.el7sat.noarch.rpm and copy it to Satellite server

3. # yum install ./tfm-rubygem-foreman-tasks-0.17.5.2-4.HOTFIXRHBZ1817728.el7sat.noarch.rpm --disableplugin=foreman-protector

4. # systemctl restart dynflowd httpd
Comment 14 wclark 2020-06-03 16:57:09 UTC 
Created attachment 1694862 [details]
FURTHER UPDATED Hotfix RPM

IMPORTANT UPDATE:

The previously provided Hotfix RPM included a Work In Progress Fix for BZ1817728 "Default task polling is too frequent at scale", which has some significant differences from the final merged version of the fix. Ordinarily, Satellite Engineering does not ship code that has not yet been in merged upstream, but after testing, an exception was made in this case to deliver an immediate fix to our customers. The fix has since undergone further revisions, so this Hotfix has been rebuilt to match the final merged version.

Please find attached the latest update to the Hotfix, as well as installation instructions for cases where the Hotfix is installed for the first time, or updating the previous Hotfix version.

INSTALLATION INSTRUCTIONS [FRESH INSTALL OF HOTFIX]:

1. Make a complete backup or snapshot of Satellite server

2. Download the attached file tfm-rubygem-foreman-tasks-0.17.5.2-5.HOTFIXRHBZ1817728.el7sat.noarch.rpm and copy it to Satellite server

3. # yum install ./tfm-rubygem-foreman-tasks-0.17.5.2-5.HOTFIXRHBZ1817728.el7sat.noarch.rpm --disableplugin=foreman-protector

4. # systemctl restart dynflowd httpd

INSTALLATION INSTRUCTIONS [UPDATING ANY EARLIER HOTFIX VERSION]

Steps 1-4, the same as above, and additionally

5. # echo "Setting.where(name: 'foreman_tasks_polling_intervals').delete_all" | foreman-rake console
Comment 15 Peter Ondrejka 2020-07-14 11:12:10 UTC 
Verified on Satellite 6.8 snap 8, the polling interval setting has been added and gets applied
Comment 18 errata-xmlrpc 2020-10-27 13:01:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.8 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4366
Note You need to log in before you can comment on or make changes to this bug.