Bug 1576103

Description of problem:

`crm_mon -s` prints "CLUSTER OK" when there are unclean (online) nodes

The '-s' switch for `crm_mon` is meant to provide a one-line output suitable for Nagios. Currently, `crm_mon -s` prints "CLUSTER OK" unless one of the following conditions is met:
  - no DC
  - at least one offline node

If there is a node in "unclean (online)" status -- e.g., with a pending/failed fence event against it -- the output will still be "CLUSTER OK". This leads an administrator to believe that the cluster is completely healthy when in fact it is not, even though all members are online.


The '-s' switch causes the print_simple_status function to execute: https://github.com/ClusterLabs/pacemaker/blob/Pacemaker-1.1.18/tools/crm_mon.c#L884-L923

There are only two calls to the `mon_warn` macro: line 828 and line 841.

The `for` loop that processes node status does not handle "unclean" state in any special way, as the print_status function does. It only checks:
  - standby
  - maintenance
  - online/offline

 831     for (gIter = data_set->nodes; gIter != NULL; gIter = gIter->next) {
 832         node_t *node = (node_t *) gIter->data;
 833 
 834         if (node->details->standby && node->details->online) {
 835             nodes_standby++;
 836         } else if (node->details->maintenance && node->details->online) {
 837             nodes_maintenance++;
 838         } else if (node->details->online) {
 839             nodes_online++;
 840         } else {
 841             mon_warn(" offline node: %s", node->details->uname);
 842         }
 843     }


So a couple of approaches to this problem, if we want to print "CLUSTER WARN" in case of any unclean nodes, would be to:
  - add a mon_warn() call in case of `node->details->unclean && node->details->online` and leave the rest as it is
  - eliminate the "offline node" mon_warn() call in favor of "unclean (online)" and "unclean (offline)" mon_warn() calls


--------------------

Version-Release number of selected component (if applicable):

pacemaker-1.1.18-11.el7.x86_64
reproducible in master


--------------------

How reproducible:

always


--------------------

Steps to Reproduce:

1. Create a GFS2 Filesystem stack (controld, clvm, Filesystem) on a two-node cluster.

2. Remove the stonith device for node 2, or remove node 2 from pcmk_host_map.

3. Unmount the GFS2 filesystem on node 2, causing a monitor failure.

4. Run `crm_mon -1` and verify that node 2 is in "UNCLEAN (online)" state.

    # crm_mon -1
    Stack: corosync
    Current DC: nwahl-rhel7-5-clus1 (version 1.1.18-11.el7-2b07d5c5a9) - partition with quorum
    Last updated: Tue May  8 16:56:22 2018
    Last change: Tue May  8 16:55:58 2018 by root via cibadmin on nwahl-rhel7-5-clus1

    2 nodes configured
    7 resources configured

    Node nwahl-rhel7-5-clus2: UNCLEAN (online)
    Online: [ nwahl-rhel7-5-clus1 ]

    Active resources:

     Clone Set: dlm-clone [dlm]
         Started: [ nwahl-rhel7-5-clus1 nwahl-rhel7-5-clus2 ]
     Clone Set: clvmd-clone [clvmd]
         Started: [ nwahl-rhel7-5-clus1 nwahl-rhel7-5-clus2 ]
     Clone Set: clusterfs-clone [clusterfs]
         clusterfs	(ocf::heartbeat:Filesystem):	FAILED nwahl-rhel7-5-clus2
         Started: [ nwahl-rhel7-5-clus1 ]
     xvm	(stonith:fence_xvm):	Started nwahl-rhel7-5-clus1

    Failed Actions:
    * clusterfs_monitor_60000 on nwahl-rhel7-5-clus2 'not running' (7): call=26, status=complete, exitreason='',
        last-rc-change='Tue May  8 16:55:40 2018', queued=0ms, exec=0ms


5. Run `crm_mon -s` and observe that it prints "CLUSTER OK".

    # crm_mon -s
    CLUSTER OK: 2 nodes online, 7 resources configured


--------------------

Actual results:

    # crm_mon -s
    CLUSTER OK: 2 nodes online, 7 resources configured


--------------------

Expected results:

    # crm_mon -s
    CLUSTER WARN: unclean (online) node: nwahl-rhel7-5-clus2


--------------------

Additional info:

N/A