Description of problem: The webserver is getting hit with calls to buglist.cgi and xmlrpc.cgi that are requesting *all* bugs in bugzilla. This causes apache to balloon to 1.1 Gigs, and we've had reports of processes hitting 6.5Gig (not confirmed if these were buglist.cgi calls) How reproducible: https://bz-web2-test.devel.redhat.com/buglist.cgi?query_format=advanced Actual results: Browser collapses under weight of returned content Expected results: Some rational limits are put in place to stop abuse of the web server. Does anyone need to retrieve over 4000 records at a time? Perhaps in xmlrpc but not in buglist. The web page could implement some sort of paging at least. Additional information: - perhaps a standalone xmlrpc server could be put in place.
Taking
Created attachment 313830 [details] Patch to limit amount of rows returned to the user in buglist.cgi and webservices (v1) Patch added to limit the number of rows returned to either the web UI or the XMLRPC query methods. The number is currently 20000 but can be configured in data/params. I am also adding extra logging lines to display the actual number of rows returned so we can look in the logs to see how often users are getting this error. Please review Dave
Created attachment 313833 [details] buglist.cgi with KBakers take Hi Dave, I thought you would have put the exit as soon as the query had run so we don't waste anymore memory on reading it into the @buglist @bugid arrays. Here's my take on it. I like having the limit set via parameter. Makes it easy to tweak
(In reply to comment #2) > The number is currently 20000 but can be configured in data/params. Hrm, I just had a thought that we may want to differentiate between buglist and xmlrpc limits. We may let larger amounts go thru xmlrpc since it doesn't go thru Template toolkit?
Created attachment 313846 [details] Patch to limit amount of rows returned to the user in buglist.cgi and webservices (v1) New patch: Added new buglistrowlimitxmlrpc param for webservice methods so that they can be tuned higher if needed. Also move the ThrowUserError() earlier since MySQL can actually count the rows selected before they are actually fetched. Please review. Dave
Pushed attachment 313846 [details] to live servers so we will see how much useful information we get.
Note that extensions should not be using incrementally higher error numbers in Bugzilla::WebService::Constants, as that will conflict with future error numbers. You should add 10000 to your error numbers.
Comment on attachment 313846 [details] Patch to limit amount of rows returned to the user in buglist.cgi and webservices (v1) Looks ok. NAK'd it on the basis of Max's comment.
Updated the fault code to be 10000 in our CVS tree. Should show up in the next scheduled update.
I'd also be interested in some input upstream on what exactly is causing your OOMs, particularly the 6GB one (normal buglist.cgi should never never get that large, even with 500,000 bugs).
I think this can be closed now as we figured out the performance issue and have remedied it. We will switch over to mod_perl in the near future once we get some other issue resolved non-specific to Bugzilla.