Bug 1422690

Summary: viewing REX job in web UI is very slow as non-admin user
Product: Red Hat Satellite Reporter: Chris Duryee <cduryee>
Component: Remote ExecutionAssignee: Ivan Necas <inecas>
Status: CLOSED ERRATA QA Contact: Peter Ondrejka <pondrejk>
Severity: high Docs Contact:
Priority: high    
Version: 6.2.7CC: bbuckingham, bkearney, cduryee, inecas, jcallaha, mhulan, mmccune
Target Milestone: UnspecifiedKeywords: Performance, PrioBumpField
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rubygem-foreman_remote_execution-0.3.0.16-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1446711 (view as bug list) Environment:
Last Closed: 2017-06-20 17:22:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1430554    
Attachments:
Description Flags
patch to skip authorizer cache in some cases none

Description Chris Duryee 2017-02-15 22:14:49 UTC 
Description of problem:

If you have a non-admin user with rights to view REX jobs, it can be very slow to view the task output. For example:

  Rendered /opt/.../app/views/job_invocations/_tab_overview.html.erb (23.2ms)
  Rendered /opt/.../app/views/job_invocations/_host_name_td.html.erb (3065.9ms)
  Rendered /opt/.../app/views/job_invocations/_host_status_td.html.erb (0.5ms)
  Rendered /opt/.../app/views/job_invocations/_host_actions_td.html.erb (3587.1ms)
  Rendered /opt/.../app/views/job_invocations/_host_name_td.html.erb (2876.5ms)
  Rendered /opt/.../app/views/job_invocations/_host_status_td.html.erb (0.5ms)
  Rendered /opt/.../app/views/job_invocations/_host_actions_td.html.erb (2717.4ms)
  Rendered /opt/.../app/views/job_invocations/_host_name_td.html.erb (3664.2ms)
  Rendered /opt/.../app/views/job_invocations/_host_status_td.html.erb (0.4ms)
  Rendered /opt/.../app/views/job_invocations/_host_actions_td.html.erb (2913.4ms)
  Rendered /opt/.../app/views/job_invocations/_host_name_td.html.erb (2749.4ms)
  Rendered /opt/.../app/views/job_invocations/_host_status_td.html.erb (0.5ms)
  Rendered /opt/.../app/views/job_invocations/_host_actions_td.html.erb (3694.2ms)
  Rendered /opt/.../app/views/job_invocations/_host_name_td.html.erb (2884.7ms)
  Rendered /opt/.../app/views/job_invocations/_host_status_td.html.erb (0.5ms)
  Rendered /opt/.../app/views/job_invocations/_host_actions_td.html.erb (2732.3ms)
  Rendered /opt/.../app/views/job_invocations/_host_name_td.html.erb (3653.0ms)
  Rendered /opt/.../app/views/job_invocations/_host_status_td.html.erb (0.5ms)
  Rendered /opt/.../app/views/job_invocations/_host_actions_td.html.erb (2917.3ms)
  Rendered /opt/.../app/views/job_invocations/_tab_hosts.html.erb (37465.4ms)
  Rendered /opt/.../app/views/job_invocations/show.html.erb within layouts/application (37502.5ms)
  Rendered layouts/_application_content.html.erb (0.2ms)
  Rendered home/_submenu.html.erb (1.4ms)
  Rendered home/_user_dropdown.html.erb (2.5ms)
Read fragment views/tabs_and_title_records-21 (0.1ms)
  Rendered home/_topbar.html.erb (6.4ms) 
  Rendered layouts/base.html.erb (7.2ms) 
Completed 200 OK in 37539ms (Views: 37366.9ms | ActiveRecord: 148.5ms)


If you are logged in as admin, this takes less than one second:

Completed 200 OK in 67ms (Views: 47.9ms | ActiveRecord: 6.9ms)


Version-Release number of selected component (if applicable): 6.2.7

Steps to Reproduce:
1. register a large number of hosts to the satellite (perhaps 40K or more)
2. view rex job. The job doesn't have to be on a large number of hosts, the example above was for 7 hosts.

Actual results: page seems to take 3-5 seconds per host


Expected results: page should be about the same speed as for admin user


Additional info: attached patch seems to help significatly
Comment 1 Chris Duryee 2017-02-15 22:16:19 UTC 
Created attachment 1250733 [details]
patch to skip authorizer cache in some cases
Comment 2 Chris Duryee 2017-02-15 22:17:48 UTC 
3 seconds per host can quickly make things unusable, marking as high severity
Comment 4 Satellite Program 2017-02-17 19:13:13 UTC 
Upstream bug assigned to cduryee
Comment 5 Satellite Program 2017-02-17 19:13:16 UTC 
Upstream bug assigned to cduryee
Comment 7 Satellite Program 2017-02-21 15:14:27 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/18554 has been resolved.
Comment 10 Chris Duryee 2017-03-07 00:16:13 UTC 
The upstream patch was reverted. I marked the attached BZ patch as obsolete, and reset back to untriaged since it no longer has a fix in-hand.
Comment 11 Ivan Necas 2017-03-09 08:24:08 UTC 
Created redmine issue http://projects.theforeman.org/issues/18851 from this bug
Comment 12 Satellite Program 2017-03-09 09:13:36 UTC 
Upstream bug assigned to inecas
Comment 13 Satellite Program 2017-03-09 09:13:40 UTC 
Upstream bug assigned to inecas
Comment 14 Satellite Program 2017-03-09 21:13:38 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/18851 has been resolved.
Comment 16 Peter Ondrejka 2017-05-30 14:42:59 UTC 
Verified with 10k hosts on satellite-6.2.10-2.0.el7sat.noarch, no significant delay experienced
Comment 18 errata-xmlrpc 2017-06-20 17:22:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1553