Red Hat Bugzilla – Bug 964955
Bugzilla application is unresponsive if read-write master is offline
Last modified: 2013-11-28 23:12:25 EST
Description of problem:
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Set "shutdownhtml" message to indicate application maintenance
2. Stop MySQL service on read-write master DB
3. Watch webserver process load go >100
Application server becomes unresponsive due to excessive load
With a shutdown message set, no DB calls should be made
Can you please help to verify this bug when its status become ON_QA? As QE has no permission to execute these steps.
(In reply to Shirley Zhou from comment #1)
> Hi, Mark
> Can you please help to verify this bug when its status become ON_QA? As QE
> has no permission to execute these steps.
You could reproduce this though.
The system behaviour is observed on the webserver whenever the database server cannot be contacted.
Using "top" on the webserver, watch the process list when connections are being made to Bugzilla in normal circumstances. You should see that processes are generally short lived and there are only a few instances of CGI processes in the process list.
[Optional] set shutdownhtml='Maintenance' in /var/www/html/bugzilla/data/params
Stop the database server configured in /var/www/html/bugzilla/localconfig.
Again watch 'top' while connections to Bugzilla are made via web sessions and XMLRPC calls. You will see that the processes hang around for a long time and that the number of CGI processes grows, as does the load on the system.
I have been unable to reproduce this. With MySQL shut down, a page loads < 1 second for regardless of whether the person has a cookie or not. I've even tried to point my db config to a host in the US (my VPS), thinking maybe localhost was to close, and got the same result.
I've spent a couple of hours trying to reproduce this issue on a VM, without success.
I tested by running 50 parallel instances of a buglist.cgi query for all open bugs in the Bugzilla product and 50 parallel instances of a show_bug.cgi query.
I ran these queries three times, once with the system running normally, once with Bugzilla shutdown, and once with Bugzilla shutdown and the mysqld service stopped.
On my low-powered VM, these queries cause a significant spike in system load in the first case and a smaller but still significant spike in the second and third cases (as apache still has to spawn processes to handle the requests when Bugzilla is in shutdown mode).
There was little difference in system load for the second and third cases, and the cgi jobs appear to be relatively shortlived, as expected, and certainly shorter lived than in the first case.
Note that my VM does not employ separate RW master and RO slave databases, though Simon believes that shouldn't be a factor in this bug.
It's also perhaps worth noting that the original report corresponds to the time of the Bugzilla 4.2 -> 4.4 upgrade, where the RW master database server failed.
As neither Simon nor myself can reproduce this bug, I'm inclined to close the report for now. If the problem is observed again, it would be useful to capture the relevant section of the apache access logs to allow analysis of the particular requests that were occuring at the time (in case it's only certain requests that misbehave when shutdown) and the possibility that some downstream system reacted badly to the shutdown by retrying requests at an unreasonable frequency.