Bug 500709 - Performance issue with error page handling on Satellite: 404.pxt (& 403, 500, 413) generates a session for all requests regardless of validity of url
Performance issue with error page handling on Satellite: 404.pxt (& 403, 500,...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Satellite 5
Classification: Red Hat
Component: Server (Show other bugs)
520
All Linux
high Severity high
: ---
: ---
Assigned To: Shannon Hughes
Preethi Thomas
:
Depends On:
Blocks: 456985
  Show dependency treegraph
 
Reported: 2009-05-13 15:10 EDT by Xixi
Modified: 2010-10-23 05:33 EDT (History)
5 users (show)

See Also:
Fixed In Version: sat530
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-09-10 16:37:01 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Xixi 2009-05-13 15:10:46 EDT
Public Bug Summary -

Description of problem:
While investigating a customer issue, it's found that Satellite has suboptimal handling of 404 pages.  Specifically, 404.pxt, such that it generates a session even for bad urls, adding unnecessary load.  In customer's case, doing a security scanning for thousands (8-9k) of bad urls via monitoring tools resulted in load spikes and unresponsive on their 5.2 satellite.

Tested in-house and talked it over with jsherrill. There are 2 kinds of 404 pages in satellite - 404.pxt for apache, and 404.jsp for java:  

/etc/rhn/satellite-httpd/conf/rhnweb.conf
...
ErrorDocument 404 /errors/404.pxt
...
<IfModule mod_jk.c>
 JkWorkersFile /etc/rhn/satellite-httpd/conf/workers.properties
 JkLogLevel info
 JkLogFile /var/log/httpd/mod_jk.log
 JkMount /rhn/* ajp13
 JkMount /rhn ajp13
 JkMount /*.do ajp13
 JkMount /*.jsp ajp13
 JkHTTPSIndicator HTTPS
</IfModule>
...

/var/lib/tomcat5/webapps/rhn/WEB-INF/web.xml
...
 <error-page>
     <error-code>404</error-code>
     <location>/WEB-INF/pages/common/errors/404.jsp</location>
 </error-page>
...

Now, both of these 404 pages are dynamic b/c they need to render the 404 page with a consistent look and feel as the rest of the site, which shouldn't be much load.  And for java, you have to be logged in to see the 404 page, otherwise you just get a login required page.  However, the problem is for apache/mod_perl, a session seems to be created by Request.pm (for non-java pages) which seems to be more of a concern regarding load/performance:

For example if I do "wget http://SATFQDN/demo/ojspext/events/globals.jsa", httpd error_log shows:
[Tue May 12 16:31:22 2009] [error] [client x.x.x.x] File does not exist: /var/www/html/demo
[Tue May 12 16:31:22 2009] [error] PXT::Request::cookie_jar (/usr/lib/perl5/site_perl/5.8.8/PXT/Request.pm:438): Generating session cookie for user: 'none', session name: 'pxt-session-cookie', value: '196707x4a6fccea044ce5592498895450324de0', expire: 'never'.

To work around this, just modify /etc/rhn/satellite-httpd/conf/rhnweb.conf so instead of /var/www/html/errors/404.pxt make it point to a static page, say /var/www/html/errors/404.html (which you have to make), by changing "ErrorDocument 404 /errors/404.pxt" to "ErrorDocument 404 /errors/404.html" and restarting satellite-httpd.

Version-Release number of selected component (if applicable):
5.2.0

How reproducible:
Always

Steps to Reproduce:
1. wget a non-java, non-existent url on Satellite (such as above).
2. look at httpd error_log and see it creates a session (see above).
3. to reproduce the load/performance issue, you will need to request thousands of bad urls via multiple processes simultaneously (see internal reproducer info).
  
Actual results:
Session created when 404.pxt is hit even for bad urls.

Expected results:
Either no session created for bad urls, or change to static 404 page.

Additional info:
Internal reproducer info and background to follow.
Comment 3 James M. Leddy 2009-05-13 15:42:24 EDT
Same for 413 and 500, I would imagine
Comment 4 Xixi 2009-05-14 14:32:26 EDT
(In reply to comment #3)
> Same for 413 and 500, I would imagine  

Right so it should really be all invalid or otherwise just requests that shouldn't have sessions generated (examples for 403 and 550 below).  So we have - 403, 404, 413 and 500 per /etc/rhn/satellite-httpd/conf/rhnweb.conf
...
ErrorDocument 403 /errors/permission.pxt
ErrorDocument 404 /errors/404.pxt
ErrorDocument 413 /errors/413.pxt
ErrorDocument 500 /errors/500.pxt
...

Updating bug summary as such.

403 -

$ wget http://SATELLITEFQDN/cgi-bin/
...
HTTP request sent, awaiting response... 403 Forbidden
11:00:52 ERROR 403: Forbidden.

httpd error_log:
[Thu May 14 14:21:18 2009] [error] [client x.x.x.x] attempt to invoke directory as script: /var/www/cgi-bin/
[Thu May 14 14:21:18 2009] [error] PXT::Request::cookie_jar (/usr/lib/perl5/site_perl/5.8.8/PXT/Request.pm:438): Generating session cookie for user: 'none', session name: 'pxt-session-cookie', value: '208699xdf455595a85cf098a918748013e2befa', expire: 'never'.

500 -

$ wget http://SATELLITEFQDN/cgi-bin/translate_key.cgi
...
HTTP request sent, awaiting response... 500 Internal Server Error
11:08:50 ERROR 500: Internal Server Error.

httpd error_log:
[Thu May 14 14:29:17 2009] [error] DBI connect('**RHN_DB_NAME**','**RHN_DB_USERNAME**',...) failed: ORA-12154: TNS:could not resolve the connect identifier specified (DBD ERROR: OCIServerAttach) at /usr/lib/perl5/site_perl/5.8.8/NOCpulse/DBRecord.pm line 106
[Thu May 14 14:29:17 2009] [error] [client x.x.x.x] Can't call method "prepare" on an undefined value at /usr/lib/perl5/site_perl/5.8.8/NOCpulse/DBRecord.pm line 137.\n
[Thu May 14 14:29:17 2009] [error] PXT::Request::cookie_jar (/usr/lib/perl5/site_perl/5.8.8/PXT/Request.pm:438): Generating session cookie for user: 'none', session name: 'pxt-session-cookie', value: '208730x6cbc43942b8bc81d8ac0175d20ae76f0', expire: 'never'.
Comment 5 Shannon Hughes 2009-05-18 18:11:57 EDT
i'll put some conditions around the session creation in our mod_perl layer. i don't want to take the static html route as that will prove difficult to maintain against the templates.
Comment 6 Shannon Hughes 2009-06-01 14:09:31 EDT
scratch comment #5. The perl template pxt files required sessions to work correctly. Since the urls in test_urls attachment are outside of the /network path we are going to set the global error docs to point to html files. Then add a Directory tag inside apache so that all urls inside of /network point to the pxt error pages we have today.
Comment 8 Shannon Hughes 2009-06-02 11:45:36 EDT
submitted static html for non session error page
Comment 11 Shannon Hughes 2009-06-03 11:09:45 EDT
vader commit

7d726b8f00efc4a7c19e344fddf9d758532da1e1
Comment 12 Shannon Hughes 2009-06-08 13:34:34 EDT
mass move to onqa
Comment 18 Brandon Perkins 2009-09-10 16:37:01 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2009-1434.html

Note You need to log in before you can comment on or make changes to this bug.