Red Hat Bugzilla – Full Text Bug Listing
|Summary:||if MM check fails, server should go into MM|
|Product:||[Other] RHQ Project||Reporter:||John Mazzitelli <mazz>|
|Component:||Core Server||Assignee:||RHQ Project Maintainer <rhq-maint>|
|Status:||CLOSED CURRENTRELEASE||QA Contact:|
|Version:||1.1||CC:||asantos, ccrouch, jshaughn|
|Fixed In Version:||Doc Type:||Bug Fix|
|Doc Text:||Story Points:||---|
|Last Closed:||2014-05-02 16:15:12 EDT||Type:||---|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
|Bug Depends On:|
Description John Mazzitelli 2008-11-07 11:39:00 EST
if the server job that checks the server MM flag fails to get to the database (i.e. gets a SQL exception) several times, it should flip to MM since the server is probably in a bad state and we'll want agents to go to another server.
Comment 1 Jay Shaughnessy 2008-11-07 12:00:45 EST
We need to discuss what the behavior should be. In general, if the db can't be reached it may be best to actually shutdown the server. Putting the server into MM is a bit of a chicken and egg problem, going into MM requires that the operation mode be updated *in* the database.
Comment 2 John Mazzitelli 2008-11-07 12:13:41 EST
in our case, we used another server in the cloud to flip the bit to MM. but the server that was put into MM didn't know it because it coudn't get to the DB to read it.
Comment 3 Jay Shaughnessy 2008-11-07 12:34:32 EST
That's an interesting scenario. But still, I think this is more an issue of what to do if the DB is hosed. If the DB is unreachable nothing is really going to work well. If we can agree on what constitutes a db failure (and very possibly that server manager job not being able to get the current operation mode is a good candidate) we may want to consider taking the server down (perhaps try to send e-mail to the rhqadmin email address). Once the DB is validated as being stable, the admin can bring the server(s) back up. If we implement the other feature about bringing up in MM then that may play well into this, if necessary.
Comment 4 Jay Shaughnessy 2008-11-11 12:13:00 EST
This is an edge case. In general failing this read from the db is indicative of a severe condition - db down, but we'll retry after delay on the chance that it's a temporary condition. If we still can't get the operation mode then force MM.
Comment 5 John Mazzitelli 2008-11-11 12:28:15 EST
we need to put this into 1.2 to avoid the problem when a server can't even determine if it should be in MM or not (when this happens, the server needs to retry that check after a pause of a few seconds and if it still fails, immediately assume the mode is MAINTENANCE)