if the server job that checks the server MM flag fails to get to the database (i.e. gets a SQL exception) several times, it should flip to MM since the server is probably in a bad state and we'll want agents to go to another server.
We need to discuss what the behavior should be. In general, if the db can't be reached it may be best to actually shutdown the server. Putting the server into MM is a bit of a chicken and egg problem, going into MM requires that the operation mode be updated *in* the database.
in our case, we used another server in the cloud to flip the bit to MM. but the server that was put into MM didn't know it because it coudn't get to the DB to read it.
That's an interesting scenario. But still, I think this is more an issue of what to do if the DB is hosed. If the DB is unreachable nothing is really going to work well. If we can agree on what constitutes a db failure (and very possibly that server manager job not being able to get the current operation mode is a good candidate) we may want to consider taking the server down (perhaps try to send e-mail to the rhqadmin email address). Once the DB is validated as being stable, the admin can bring the server(s) back up. If we implement the other feature about bringing up in MM then that may play well into this, if necessary.
This is an edge case. In general failing this read from the db is indicative of a severe condition - db down, but we'll retry after delay on the chance that it's a temporary condition. If we still can't get the operation mode then force MM.
we need to put this into 1.2 to avoid the problem when a server can't even determine if it should be in MM or not (when this happens, the server needs to retry that check after a pause of a few seconds and if it still fails, immediately assume the mode is MAINTENANCE)
This bug was previously known as http://jira.rhq-project.org/browse/RHQ-1082 This bug relates to RHQ-921