Bug 1577085

Description of problem:

Customer has raised the concern that the output and return codes of `crm_mon -s` do not always accurately reflect the status of the cluster.

`crm_mon -s` is meant to provide a one-line output suitable for Nagios. Per the guidelines here (https://www.monitoring-plugins.org/doc/guidelines.html#AEN78), Nagios plugins can return one of four statuses: OK, Warning, Critical, or Unknown.


Excerpts from the customer:

    Bugzilla 1576103... looks like it says that both unclean (online)
    and unclean (offline) would be warning states.  Currently offline
    is critical, should the bugzilla be reworded such that we are
    recommending unclean (online) be warning and unclean (offline) be
    a critical error? ...a single system being down in a two node cluster
    would result and lack of redundancy with imminent failure is just
    around the corner and would probably step up to the level of
    critical.) Though maybe some kind of mathematical equation that if
    greater than or equal to fifty percent of the nodes are offline it
    would go from warning to critical as having one offline node in a
    cluster leaves you in a position of imminent disaster, however in a
    three or more node cluster having one unclean offline node would
    still only be a warning level as another node could fail before it
    becomes critical. This math may be over simplistic but because maybe
    in a 10 node cluster having 4 nodes offline might cause a
    unacceptable slow down.      

    New potential consideration for the simple output.
    ...I would think that ["no DC"] would be a candidate to be an
    "Unknown" as it might be a configuration issue like maybe you
    running the report on a box that doesn't yet have a cluster
    configured. ...Regardless of whether I am right or wrong on how the
    prior works I would think that the ideal simple output would have
    at least one unknown result for when the script is somehow added to
    a box that does not have either cluster installed on it, to
    returning an unknown result along the lines of "Can't find cluster
    software, some specific application, config file,…,  or a cluster 
    configuration"


--------------------

Version-Release number of selected component (if applicable):

pacemaker-1.1.18-11
master


--------------------

How reproducible:

Always


--------------------

Steps to Reproduce:

Run `crm_mon -s` with varying degrees of resource cleanliness and numbers of offline nodes. Exact details TBD.


--------------------

Actual results:

Output and return codes may be too nonspecific. For example, "OK" and "Warning" are the only options.


--------------------

Expected results:

Exact details TBD. Output and return codes accurately reflect status of cluster in alignment with Nagios plugin guidelines. For example, certain conditions may qualify as a critical state.


--------------------

Additional info:

Related to Bug 1576103 - `crm_mon -s` prints "CLUSTER OK" when there are unclean (online) nodes