Bug 458223

Summary:

mod_perl & buglist queries OOM apache

Product:

[Community] Bugzilla

Reporter:

Kevin Baker <kbaker>

Component:

Query/Bug List

Assignee:

David Lawrence <dkl>

Status:

CLOSED CURRENTRELEASE

QA Contact:

Severity:

medium

Docs Contact:

Priority:

medium

Version:

3.2

CC:

ineilsen, mkanat, mschick

Target Milestone:

---

Target Release:

---

Hardware:

All

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2008-11-24 20:31:58 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
Patch to limit amount of rows returned to the user in buglist.cgi and webservices (v1)	none
buglist.cgi with KBakers take	none
Patch to limit amount of rows returned to the user in buglist.cgi and webservices (v1)	kbaker: review-

Description Kevin Baker 2008-08-07 03:33:55 UTC

Description of problem:

The webserver is getting hit with calls to buglist.cgi and xmlrpc.cgi that are requesting *all* bugs in bugzilla. This causes apache to balloon to 1.1 Gigs, and we've had reports of processes hitting 6.5Gig (not confirmed if these were buglist.cgi calls)


How reproducible:

https://bz-web2-test.devel.redhat.com/buglist.cgi?query_format=advanced
  
Actual results:

Browser collapses under weight of returned content

Expected results:

Some rational limits are put in place to stop abuse of the web server. Does anyone need to retrieve over 4000 records at a time? Perhaps in xmlrpc but not in buglist. The web page could implement some sort of paging at least. 

Additional information:

- perhaps a standalone xmlrpc server could be put in place.

Comment 1 David Lawrence 2008-08-07 21:58:56 UTC

Taking

Comment 2 David Lawrence 2008-08-08 18:36:01 UTC

Created attachment 313830 [details]
Patch to limit amount of rows returned to the user in buglist.cgi and webservices (v1)

Patch added to limit the number of rows returned to either the web UI or the XMLRPC query methods.

The number is currently 20000 but can be configured in data/params.

I am also adding extra logging lines to display the actual number of rows returned so we can look in the logs to see how often users are getting this error.

Please review
Dave

Comment 3 Kevin Baker 2008-08-08 18:57:39 UTC

Created attachment 313833 [details]
buglist.cgi with KBakers take 

Hi Dave,

I thought you would have put the exit as soon as the query had run so we don't waste anymore memory on reading it into the @buglist @bugid arrays. Here's my take on it.

I like having the limit set via parameter. Makes it easy to tweak

Comment 4 Kevin Baker 2008-08-08 18:59:28 UTC

(In reply to comment #2)

> The number is currently 20000 but can be configured in data/params.

Hrm, I just had a thought that we may want to differentiate between buglist and xmlrpc limits. We may let larger amounts go thru xmlrpc since it doesn't go thru Template toolkit?

Comment 5 David Lawrence 2008-08-08 20:13:21 UTC

Created attachment 313846 [details]
Patch to limit amount of rows returned to the user in buglist.cgi and webservices (v1)

New patch:

Added new buglistrowlimitxmlrpc param for webservice methods so that they can be tuned higher if needed.

Also move the ThrowUserError() earlier since MySQL can actually count the rows selected before they are actually fetched.

Please review.
Dave

Comment 6 David Lawrence 2008-08-08 20:56:40 UTC

Pushed attachment 313846 [details] to live servers so we will see how much useful information we get.

Comment 7 Max Kanat-Alexander 2008-09-03 23:51:38 UTC

Note that extensions should not be using incrementally higher error numbers in Bugzilla::WebService::Constants, as that will conflict with future error numbers. You should add 10000 to your error numbers.

Comment 8 Kevin Baker 2008-09-04 15:20:04 UTC

Comment on attachment 313846 [details]
Patch to limit amount of rows returned to the user in buglist.cgi and webservices (v1)

Looks ok. NAK'd it on the basis of Max's comment.

Comment 9 David Lawrence 2008-09-04 18:11:42 UTC

Updated the fault code to be 10000 in our CVS tree. Should show up in the next scheduled update.

Comment 10 Max Kanat-Alexander 2008-09-04 22:33:24 UTC

I'd also be interested in some input upstream on what exactly is causing your OOMs, particularly the 6GB one (normal buglist.cgi should never never get that large, even with 500,000 bugs).

Comment 11 David Lawrence 2008-11-24 20:31:58 UTC

I think this can be closed now as we figured out the performance issue and have remedied it. We will switch over to mod_perl in the near future once we get some other issue resolved non-specific to Bugzilla.