Bug 1576103 - `crm_mon -s` prints "CLUSTER OK" when there are unclean (online) nodes
Summary: `crm_mon -s` prints "CLUSTER OK" when there are unclean (online) nodes
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: pacemaker
Version: 8.0
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: pre-dev-freeze
: ---
Assignee: Ken Gaillot
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-08 21:00 UTC by Reid Wahl
Modified: 2021-05-02 14:15 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Feature Request
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3441221 0 None None None 2018-05-12 00:50:40 UTC

Internal Links: 1577085

Description Reid Wahl 2018-05-08 21:00:20 UTC
Description of problem:

`crm_mon -s` prints "CLUSTER OK" when there are unclean (online) nodes

The '-s' switch for `crm_mon` is meant to provide a one-line output suitable for Nagios. Currently, `crm_mon -s` prints "CLUSTER OK" unless one of the following conditions is met:
  - no DC
  - at least one offline node

If there is a node in "unclean (online)" status -- e.g., with a pending/failed fence event against it -- the output will still be "CLUSTER OK". This leads an administrator to believe that the cluster is completely healthy when in fact it is not, even though all members are online.


The '-s' switch causes the print_simple_status function to execute: https://github.com/ClusterLabs/pacemaker/blob/Pacemaker-1.1.18/tools/crm_mon.c#L884-L923

There are only two calls to the `mon_warn` macro: line 828 and line 841.

The `for` loop that processes node status does not handle "unclean" state in any special way, as the print_status function does. It only checks:
  - standby
  - maintenance
  - online/offline

 831     for (gIter = data_set->nodes; gIter != NULL; gIter = gIter->next) {
 832         node_t *node = (node_t *) gIter->data;
 833 
 834         if (node->details->standby && node->details->online) {
 835             nodes_standby++;
 836         } else if (node->details->maintenance && node->details->online) {
 837             nodes_maintenance++;
 838         } else if (node->details->online) {
 839             nodes_online++;
 840         } else {
 841             mon_warn(" offline node: %s", node->details->uname);
 842         }
 843     }


So a couple of approaches to this problem, if we want to print "CLUSTER WARN" in case of any unclean nodes, would be to:
  - add a mon_warn() call in case of `node->details->unclean && node->details->online` and leave the rest as it is
  - eliminate the "offline node" mon_warn() call in favor of "unclean (online)" and "unclean (offline)" mon_warn() calls


--------------------

Version-Release number of selected component (if applicable):

pacemaker-1.1.18-11.el7.x86_64
reproducible in master


--------------------

How reproducible:

always


--------------------

Steps to Reproduce:

1. Create a GFS2 Filesystem stack (controld, clvm, Filesystem) on a two-node cluster.

2. Remove the stonith device for node 2, or remove node 2 from pcmk_host_map.

3. Unmount the GFS2 filesystem on node 2, causing a monitor failure.

4. Run `crm_mon -1` and verify that node 2 is in "UNCLEAN (online)" state.

    # crm_mon -1
    Stack: corosync
    Current DC: nwahl-rhel7-5-clus1 (version 1.1.18-11.el7-2b07d5c5a9) - partition with quorum
    Last updated: Tue May  8 16:56:22 2018
    Last change: Tue May  8 16:55:58 2018 by root via cibadmin on nwahl-rhel7-5-clus1

    2 nodes configured
    7 resources configured

    Node nwahl-rhel7-5-clus2: UNCLEAN (online)
    Online: [ nwahl-rhel7-5-clus1 ]

    Active resources:

     Clone Set: dlm-clone [dlm]
         Started: [ nwahl-rhel7-5-clus1 nwahl-rhel7-5-clus2 ]
     Clone Set: clvmd-clone [clvmd]
         Started: [ nwahl-rhel7-5-clus1 nwahl-rhel7-5-clus2 ]
     Clone Set: clusterfs-clone [clusterfs]
         clusterfs	(ocf::heartbeat:Filesystem):	FAILED nwahl-rhel7-5-clus2
         Started: [ nwahl-rhel7-5-clus1 ]
     xvm	(stonith:fence_xvm):	Started nwahl-rhel7-5-clus1

    Failed Actions:
    * clusterfs_monitor_60000 on nwahl-rhel7-5-clus2 'not running' (7): call=26, status=complete, exitreason='',
        last-rc-change='Tue May  8 16:55:40 2018', queued=0ms, exec=0ms


5. Run `crm_mon -s` and observe that it prints "CLUSTER OK".

    # crm_mon -s
    CLUSTER OK: 2 nodes online, 7 resources configured


--------------------

Actual results:

    # crm_mon -s
    CLUSTER OK: 2 nodes online, 7 resources configured


--------------------

Expected results:

    # crm_mon -s
    CLUSTER WARN: unclean (online) node: nwahl-rhel7-5-clus2


--------------------

Additional info:

N/A

Comment 2 Ken Gaillot 2018-05-08 22:14:30 UTC
Unfortunately we are saturated for the 7.6 time frame, so I am proposing this for 7.7.

For an interim, unsupported solution, this community plugin may be of interest (though it might require some editing for any particular environment): https://exchange.nagios.org/directory/Plugins/Clustering-and-High-2DAvailability/Check-CRM/details

Comment 3 Ken Gaillot 2019-03-27 20:57:14 UTC
Due to this not making the 7.7 time frame, I'm moving it to RHEL 8 only, as RHEL 7 will only be getting bug fixes from this point.

The approach I'm leaning to here is creating a new tool (maybe called crm_check) that would solely be a monitoring plugin, since it doesn't really overlap with crm_mon much. We would likely handle Bug 1577085 at the same time.

Unfortunately due to developer constraints, I cannot commit to a release time frame.


Note You need to log in before you can comment on or make changes to this bug.