Bug 1276443

I have 2 cases opened in relation to this and bug 1269509, both of which are regarding timeouts and patching performance in 6.1.3.

Case 01531872 is related to this timeout issue and 01530408 is related as this same thing applies to repo synch's.  What I've found to be the underlying problem (which is also mentioned in bug 1269509) is the reliance on timeouts being a bad idea.  Katello / Foreman provide the GUI and Pulp runs the work in these cases, it's a really bad design, IMHO, to have katello tell pulp what to do and then simply forget about it/say it failed if the task isn't completed soon enough.  To add insult to injury here, the task isn't cancelled in pulp when Katello times out so, pulp will still get around to doing the work ... eventually.

These 2 products need to talk together to provide the end user a much better idea of what's actually going on.  After 2 weeks of digging into this, here are the recommendations I'd make ... hope you guys are receptive to ideas:

1.  Add a pulp status page that will show the queue depth and at least what tasks are currently processing.  Adding the ability to prioritize tasks in pulp, batch cancel in pulp and drill down to see what tasks are what in the queue would be awesome. As critical as pulp is, it cannot be so detached from the GUI.  (EX: ALWAYS move patching activities to the front of the queue ahead of repo synch's)

2. Add some logic into Katello or pulp to not queue duplicate tasks.  If a repo is synching now, don't add another task to synch it again until after this synch has completed.  Same for client updates, etc.  All this does is allows the queue to back up to unmanageable proportions and slow the whole system down.

3. Allow a "max queue depth" setting for katello and/or pulp.  don't accept more work if there's a big backlog.

4. Repo synch's ... check for changes, if there are none, don't regenerate metadata or synch the capsules, after all, nothing's changed.  All this does is adds VERY long running tasks into the katello and pulp queues.  This is how I ended up with 11,000 tasks to cancel in pulp.

Please see this comment here for HOTFIX instructions on how to apply large # of errata with Satellite 6:

https://bugzilla.redhat.com/show_bug.cgi?id=1269509#c18

VERIFIED:

# rpm -qa | grep foreman
foreman-compute-1.7.2.48-1.el7sat.noarch
ruby193-rubygem-foreman-redhat_access-0.2.4-1.el7sat.noarch
rubygem-hammer_cli_foreman_discovery-0.0.1.10-1.el7sat.noarch
ruby193-rubygem-foreman_bootdisk-4.0.2.13-1.el7sat.noarch
ruby193-rubygem-foreman_docker-1.2.0.24-1.el7sat.noarch
rubygem-hammer_cli_foreman_docker-0.0.3.10-1.el7sat.noarch
foreman-debug-1.7.2.48-1.el7sat.noarch
foreman-postgresql-1.7.2.48-1.el7sat.noarch
ruby193-rubygem-foreman_openscap-0.3.2.10-1.el7sat.noarch
ruby193-rubygem-foreman_hooks-0.3.7-2.el7sat.noarch
foreman-gce-1.7.2.48-1.el7sat.noarch
ruby193-rubygem-foreman_discovery-2.0.0.22-1.el7sat.noarch
rubygem-hammer_cli_foreman_tasks-0.0.3.5-1.el7sat.noarch
puppet-foreman_scap_client-0.3.3-9.el7sat.noarch
foreman-selinux-1.7.2.17-1.el7sat.noarch
foreman-vmware-1.7.2.48-1.el7sat.noarch
ruby193-rubygem-foreman_gutterball-0.0.1.9-1.el7sat.noarch
foreman-proxy-1.7.2.6-1.el7sat.noarch
ibm-x3655-04.ovirt.rhts.eng.bos.redhat.com-foreman-proxy-client-1.0-1.noarch
foreman-discovery-image-3.0.5-3.el7sat.noarch
foreman-1.7.2.48-1.el7sat.noarch
ruby193-rubygem-foreman-tasks-0.6.15.7-1.el7sat.noarch
rubygem-hammer_cli_foreman_bootdisk-0.1.2.7-1.el7sat.noarch
foreman-libvirt-1.7.2.48-1.el7sat.noarch
ibm-x3655-04.ovirt.rhts.eng.bos.redhat.com-foreman-client-1.0-1.noarch
ibm-x3655-04.ovirt.rhts.eng.bos.redhat.com-foreman-proxy-1.0-1.noarch
foreman-ovirt-1.7.2.48-1.el7sat.noarch
rubygem-hammer_cli_foreman-0.1.4.14-1.el7sat.noarch


steps:
1. Select multiple errata(s) to create a transaction that will take at least 120 seconds to apply
2. Apply selected errata


 Id: e41062bf-bfd1-52db-c67c-f623cff8egg7
Label: Actions::Katello::System::Erratum::Install
Name: Install erratum
Owner: admin
Started at: 2015-12-07 11:52:07 UTC
Ended at: 2015-12-07 11:57:10 UTC
State: stopped
Result: success
Params: RHSA-2015:2504, RHBA-2015:2014, RHBA-2015:2018, RHBA-2015:2006, RHBA-2015:2011; system 'ibm-hs22-02.lab.bos.redhat.com'; organization 'Default Organization' 


perl-IO-Compress-Base-2.021-141.el6_7.1.x86_64
libreport-python-2.0.9-25.el6_7.x86_64
1:perl-Module-Pluggable-3.90-141.el6_7.1.x86_64
3:perl-version-0.77-141.el6_7.1.x86_64
4:perl-devel-5.10.1-141.el6_7.1.x86_64
4:perl-5.10.1-141.el6_7.1.x86_64
1:perl-Compress-Raw-Zlib-2.021-141.el6_7.1.x86_64
libreport-plugin-kerneloops-2.0.9-25.el6_7.x86_64
perl-CGI-3.51-141.el6_7.1.x86_64
libreport-plugin-mailx-2.0.9-25.el6_7.x86_64
zip-3.0-1.el6_7.1.x86_64
1:perl-Pod-Simple-3.13-141.el6_7.1.x86_64
1:perl-Pod-Escapes-1.04-141.el6_7.1.x86_64
libreport-compat-2.0.9-25.el6_7.x86_64
libreport-plugin-rhtsupport-2.0.9-25.el6_7.x86_64
selinux-policy-3.7.19-279.el6_7.7.noarch
4:perl-libs-5.10.1-141.el6_7.1.x86_64
libreport-2.0.9-25.el6_7.x86_64
perl-Test-Simple-0.92-141.el6_7.1.x86_64
libreport-cli-2.0.9-25.el6_7.x86_64
perl-ExtUtils-MakeMaker-6.55-141.el6_7.1.x86_64
1:perl-ExtUtils-ParseXS-2.2003.0-141.el6_7.1.x86_64
libreport-plugin-logger-2.0.9-25.el6_7.x86_64
perl-Test-Harness-3.17-141.el6_7.1.x86_64
grep-2.20-3.el6_7.1.x86_64
libreport-plugin-reportuploader-2.0.9-25.el6_7.x86_64
selinux-policy-targeted-3.7.19-279.el6_7.7.noarch
perl-IO-Compress-Zlib-2.021-141.el6_7.1.x86_64
perl-Compress-Zlib-2.021-141.el6_7.1.x86_64
json-c-0.11-12.el6.x86_64
libreport-filesystem-2.0.9-25.el6_7.x86_64
satyr-0.16-2.el6.x86_64
augeas-libs-1.0.0-10.el6.x86_64

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:2622