Bug 1817728
| Summary: | Default task polling is too frequent at scale | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Mike McCune <mmccune> | ||||||||||
| Component: | Tasks Plugin | Assignee: | Adam Ruzicka <aruzicka> | ||||||||||
| Status: | CLOSED ERRATA | QA Contact: | Peter Ondrejka <pondrejk> | ||||||||||
| Severity: | medium | Docs Contact: | |||||||||||
| Priority: | unspecified | ||||||||||||
| Version: | 6.6.0 | CC: | aruzicka, egolov, hyu, jdickers, ktordeur, momran, pmoravec, wclark | ||||||||||
| Target Milestone: | 6.8.0 | Keywords: | Triaged | ||||||||||
| Target Release: | Unused | ||||||||||||
| Hardware: | Unspecified | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | tfm-rubygem-foreman-tasks-1.1.1 | Doc Type: | If docs needed, set a value | ||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | |||||||||||||
| : | 1832587 (view as bug list) | Environment: | |||||||||||
| Last Closed: | 2020-10-27 13:01:14 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Attachments: |
|
||||||||||||
Created redmine issue https://projects.theforeman.org/issues/29423 from this bug Upstream bug assigned to aruzicka Upstream bug assigned to aruzicka Created attachment 1679728 [details]
tfm-rubygem-foreman-tasks hotfix RPM for Satellite 6.7.0
HOTFIX is available for Satellite 6.7.0 INSTALLATION INSTRUCTIONS: 1. Make a complete backup or snapshot of Satellite server 2. Download the attached file tfm-rubygem-foreman-tasks-0.17.5.2-2.HOTFIXRHBZ1817728.el7sat.noarch.rpm and copy it to Satellite server 3. # yum install ./tfm-rubygem-foreman-tasks-0.17.5.2-2.HOTFIXRHBZ1817728.el7sat.noarch.rpm --disableplugin=foreman-protector 4. # systemctl restart dynflowd httpd Created attachment 1682631 [details] UPDATED tfm-rubygem-foreman-tasks hotfix RPM This also includes the hotfix for https://bugzilla.redhat.com/show_bug.cgi?id=1824931 The filename has changed but installation instructions are otherwise the same as before Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/29423 has been resolved. Created attachment 1694147 [details]
UPDATED hotfix RPM
INSTALLATION INSTRUCTIONS:
1. Make a complete backup or snapshot of Satellite server
2. Download the attached file tfm-rubygem-foreman-tasks-0.17.5.2-4.HOTFIXRHBZ1817728.el7sat.noarch.rpm and copy it to Satellite server
3. # yum install ./tfm-rubygem-foreman-tasks-0.17.5.2-4.HOTFIXRHBZ1817728.el7sat.noarch.rpm --disableplugin=foreman-protector
4. # systemctl restart dynflowd httpd
Created attachment 1694862 [details] FURTHER UPDATED Hotfix RPM IMPORTANT UPDATE: The previously provided Hotfix RPM included a Work In Progress Fix for BZ1817728 "Default task polling is too frequent at scale", which has some significant differences from the final merged version of the fix. Ordinarily, Satellite Engineering does not ship code that has not yet been in merged upstream, but after testing, an exception was made in this case to deliver an immediate fix to our customers. The fix has since undergone further revisions, so this Hotfix has been rebuilt to match the final merged version. Please find attached the latest update to the Hotfix, as well as installation instructions for cases where the Hotfix is installed for the first time, or updating the previous Hotfix version. INSTALLATION INSTRUCTIONS [FRESH INSTALL OF HOTFIX]: 1. Make a complete backup or snapshot of Satellite server 2. Download the attached file tfm-rubygem-foreman-tasks-0.17.5.2-5.HOTFIXRHBZ1817728.el7sat.noarch.rpm and copy it to Satellite server 3. # yum install ./tfm-rubygem-foreman-tasks-0.17.5.2-5.HOTFIXRHBZ1817728.el7sat.noarch.rpm --disableplugin=foreman-protector 4. # systemctl restart dynflowd httpd INSTALLATION INSTRUCTIONS [UPDATING ANY EARLIER HOTFIX VERSION] Steps 1-4, the same as above, and additionally 5. # echo "Setting.where(name: 'foreman_tasks_polling_intervals').delete_all" | foreman-rake console Verified on Satellite 6.8 snap 8, the polling interval setting has been added and gets applied Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Satellite 6.8 release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:4366 |
The default polling back-off algorithm can still cause an overload of polling during long running tasks: def poll_intervals [0.5, 1, 2, 4, 8, 16] end when there are large numbers of tasks, you can get 'storms' of status checks during long running tasks. Updated and expanded array has shown improvement in working conditions when allowing longer tasks a greater time period of checking in: [0.5, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024] We are going to update the default array to the above value and offer a configuration setting so it can be tuned to specific environments.