Bug 1275183 - Timeout in errata calculation with multiple concurrent ContentViews containing RedHat repos and Date Filter
Timeout in errata calculation with multiple concurrent ContentViews containin...
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Errata Management (Show other bugs)
6.1.3
All Linux
high Severity medium (vote)
: 6.2.2
: --
Assigned To: Justin Sherrill
Katello QA List
: PrioBumpPM, Triaged
: 1161639 1172556 (view as bug list)
Depends On:
Blocks: 1122832 1317008
  Show dependency treegraph
 
Reported: 2015-10-26 04:45 EDT by Peter Vreman
Modified: 2016-09-14 17:03 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-09-14 17:03:07 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Task export with timeout of errata calculation (3.07 MB, application/x-gzip)
2015-10-26 04:45 EDT, Peter Vreman
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2135931 None None None 2016-03-07 06:43 EST

  None (edit)
Description Peter Vreman 2015-10-26 04:45:27 EDT
Created attachment 1086414 [details]
Task export with timeout of errata calculation

Description of problem:
When publishing multiple ContentViews containing the RedHat repositories there is a timeout in the errata claus generation

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Sync RedHat repos of 6,5,6.6,6.7,7.1 releases with at least the following repos:
   - Kickstart
   - Server
   - Optional
   - RHSCL
   - Oracle Java
   - Sat6.1 Tools
2. Create for each RedHat minor a ContentView
   - E.g. CV-RHEL6.5, CV-RHEL6.6, CV-RHEL6.7
3. Add to each Content View the RedHat repositories of that release
3. Configure on each repository a dDate Filters
   - Include all Packages without Errata
   - Include by date everything upto day 2015.10.26
4. Start publishing all repositories in parallel
   Use Hammer with --async or in multiple terminals. Clicking and waiting in the UI might not be concurrent enough

Actual results:
timeouts happening

Expected results:
publish successful

Additional info:
Comment 2 Peter Vreman 2015-12-02 06:03:30 EST
I have applied the recommendation "if you're getting "Request Timeout" in the sub-tasks then the work around is as follows"
- Edit /etc/foreman/plugins/katello.yaml and set rest_client_timeout: 3600 (default is 120)
- katello-service restart

Now i can publish 3 CVs with 6 RedHat channels at the same time without a Rest timeout. See below that it still worked even that a request took 17 Minutes and an other took 11 Minutes to complete:

build	02-Dec-2015 10:08:54	2015-12-02 10:08:54 [DEBUG]Requesting GET '[KT]content_views' ("organization_id":3,"name":"product-HOIRHEL-6.6-d20151201")
build	02-Dec-2015 10:08:56	2015-12-02 10:08:56 [INFO ]Publishing Content View 'product-HOIRHEL-6.6-d20151201' to 'Library'
build	02-Dec-2015 10:08:56	2015-12-02 10:08:56 [DEBUG]Requesting POST '[KT]content_views/105/publish' ()
build	02-Dec-2015 10:25:31	2015-12-02 10:25:31 [INFO ]Waiting for 1 Tasks
build	02-Dec-2015 10:25:31	2015-12-02 10:25:31 [INFO ][TASKS]Waiting for 1 tasks, completed 0 tasks, elapsed 0 minutes, waiting 0 seconds
build	02-Dec-2015 10:25:31	2015-12-02 10:25:31 [DEBUG][TASKS]Requesting POST '[TASKS]tasks/bulk_search' ("searches":[{"type":"task","task_id":"88d17eb1-4d94-48b1-94ab-7c7be953e97e"}])

build	02-Dec-2015 09:36:57	2015-12-02 09:36:57 [DEBUG]Requesting GET '[KT]content_views' ("organization_id":3,"name":"product-HOIRHEL-6Server-d20151201")
build	02-Dec-2015 09:37:00	2015-12-02 09:37:00 [INFO ]Publishing Content View 'product-HOIRHEL-6Server-d20151201' to 'Library'
build	02-Dec-2015 09:37:00	2015-12-02 09:37:00 [DEBUG]Requesting POST '[KT]content_views/106/publish' ()
build	02-Dec-2015 09:48:53	2015-12-02 09:48:53 [INFO ]Waiting for 1 Tasks
build	02-Dec-2015 09:48:53	2015-12-02 09:48:53 [INFO ][TASKS]Waiting for 1 tasks, completed 0 tasks, elapsed 0 minutes, waiting 0 seconds
build	02-Dec-2015 09:48:53	2015-12-02 09:48:53 [DEBUG][TASKS]Requesting POST '[TASKS]tasks/bulk_search' ("searches":[{"type":"task","task_id":"c32968f5-8e05-4068-a1e0-9657e1f277a7"}])

Please include the setting of this rest_client_timeout from the API. Currently it can only be set manual and is removed again in the next upgrade
Comment 3 Peter Vreman 2015-12-02 06:06:39 EST
Logging fragment of upgrade to show that the upgrade is removing it:

[DEBUG 2015-11-30 13:23:19 main]  /Stage[main]/Katello::Config/File[/usr/share/foreman/bundler.d/katello.rb]: The container Class[Katello::Config] will propagate
 my refresh event
[DEBUG 2015-11-30 13:23:19 main]  /Stage[main]/Katello::Config/File[/usr/share/foreman/bundler.d/katello.rb]: The container Class[Katello::Config] will propagate
 my refresh event
[DEBUG 2015-11-30 13:23:19 main]  Executing 'diff -u /etc/foreman/plugins/katello.yaml /tmp/puppet-file20151130-3834-1y7ig9o-0'
[ WARN 2015-11-30 13:23:20 main]  /Stage[main]/Katello::Config/File[/etc/foreman/plugins/katello.yaml]/content:
[ INFO 2015-11-30 13:23:20 main] --- /etc/foreman/plugins/katello.yaml  2015-11-30 13:15:17.531078004 +0000
[ INFO 2015-11-30 13:23:20 main] +++ /tmp/puppet-file20151130-3834-1y7ig9o-0    2015-11-30 13:23:19.421038148 +0000
[ INFO 2015-11-30 13:23:20 main] @@ -13,7 +13,7 @@
[ INFO 2015-11-30 13:23:20 main]
[ INFO 2015-11-30 13:23:20 main]  common:
[ INFO 2015-11-30 13:23:20 main]
[ INFO 2015-11-30 13:23:20 main] -  rest_client_timeout: 3600
[ INFO 2015-11-30 13:23:20 main] +  rest_client_timeout: 120
[ INFO 2015-11-30 13:23:20 main]
[ INFO 2015-11-30 13:23:20 main]    post_sync_url: https://localhost/katello/api/v2/repositories/sync_complete?token=M3g6xdbsJkUgpNkQ5H9emr2f77Di4nKA
[ INFO 2015-11-30 13:23:20 main]
[ INFO 2015-11-30 13:23:20 main]  /Stage[main]/Katello::Config/File[/etc/foreman/plugins/katello.yaml]: Filebucketed /etc/foreman/plugins/katello.yaml to puppet with sum 533bc6fc4aee6b580589ec3500bb039e
Comment 4 Peter Vreman 2016-03-07 04:50:17 EST
*** Bug 1172556 has been marked as a duplicate of this bug. ***
Comment 8 Peter Vreman 2016-06-16 08:12:22 EDT
This is fixed in Sat6.2Beta2, the default has been increased to 3600:

[crash] root@li-lc-1578:~# grep rest_client_timeout /usr/share/katello-installer-base/modules/katello/templates/katello.yaml.erb
  :rest_client_timeout: 3600

After installing scenario 'satellite'

[crash] root@li-lc-1578:~# grep rest_client_timeout /usr/share/foreman/config/settings.plugins.d/katello.yaml
  :rest_client_timeout: 3600
Comment 9 Justin Sherrill 2016-07-22 16:15:14 EDT
*** Bug 1161639 has been marked as a duplicate of this bug. ***
Comment 11 Justin Sherrill 2016-07-26 15:24:34 EDT
It appears that this was moved from a pulp query (in 6.1) to a db query in 6.2.  Thus the rest client timeout isn't actually relevant anymore and it seems to only be taking a few seconds to initiate the publish task.  

moving to on_qa for verification
Comment 12 jcallaha 2016-09-12 16:34:03 EDT
Verified in Satellite 6.2.2 Snap 1.1

Followed the steps outlined in comment #1. Kicked off all content view publishes in rapid succession.

[root@rhsm-qe-1 ~]# for i in {9..16}; do hammer -u admin -p changeme content-view publish --id $i --async --organization-id 1; done;
Content view is being published with task 6e77516b-8fe2-4c61-b296-2d5e59cd9856
Content view is being published with task 9c9d0d79-c1e6-4812-aaf1-b9104b8252b9
Content view is being published with task 587019b2-dbbc-43c9-82b0-91fa1dcfad5c
Content view is being published with task b8005eab-d283-4adc-9a85-be96248386a0
Content view is being published with task f1f11e50-e5c6-4a3b-ade4-bb1912590c12
Content view is being published with task 48a15777-ed34-4cdb-941c-173f436b1610
Content view is being published with task f58876e9-efa3-40b1-bd3a-6d1630d7471d
Content view is being published with task be69e830-8a3d-42b5-b002-b5f6107d41af

All publishes completed successfully. No timeouts seen in any logs.
Comment 14 errata-xmlrpc 2016-09-14 17:03:07 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1885

Note You need to log in before you can comment on or make changes to this bug.