Description of problem:
Weekly storage cluster maintenance or running storage cluster maintenance manually can put a storage node into what appears to be an unrecoverable maintenance state if its agent is offline due to a synchronized restart.
For example, if agent processes are auto-restarted once a week, this can potentially correspond with the storage cluster auto maintenance job which also runs once a week. The result is that the storage node is left in an operation mode of MAINTENANCE and therefore has its cluster status reported as DOWN.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Install and start JBoss ON 3.2 system.
2. Install a second agent and storage node.
3. Verify that both storage nodes are in inventory and are UP/NORMAL.
4. Shutdown one of the agents.
5. Invoke the following JBoss ON CLI command:
./rhq-cli.sh -u rhqadmin -p rhqadmin -c 'StorageNodeManager.runClusterMaintenance()'
Storage node running on the agent that was unavailable has its cluster status reported as DOWN and its operation mode indicates MAINTENANCE.
No error or bad state associated with the node.
Although it is understandable that maintenance can not complete while the agent is unavailable, this situation should be temporary. As soon as the agent comes back online, cluster maintenance should continue. In other words, the error state should only be reported/reflected while the agent is down.
The fact that the node is stuck in MAINTENANCE also seems to indicate a desegregated cluster. Auto maintenance shouldn't cause such situations.
Bumping the target release due to time constraints. Work has been started though in the storage_workflow branch.
JBoss ON is coming to the end of its product life cycle. For more information regarding this transition, see https://access.redhat.com/articles/3827121.
This bug report/request is being closed. If you feel this issue should not be closed or requires further review, please create a new bug report against the latest supported JBoss ON 3.3 version.