Bug 1221680

Created attachment 1025457 [details]
'pcs cluster report' after several reboots

Two things:

> have-watchdog: false

This implies that sbd is not running or not properly configured.
This is backed up by:

>  Active: inactive (dead)

Probably because you've set 

> SBD_OPTS=-W

which disables the system watchdog. Don't do that.


Also:

> controld(dlm)[4556]:	2015/05/14_16:03:15 ERROR: Uncontrolled lockspace exists, system must reboot. Executing suicide fencing

This is why the node is rebooting.
Make sure the cluster is working before adding the dlm.

Created attachment 1033837 [details]
pcs cluster report

This report is taken after both nodes are rebooted by sbd.

I deleted stonith devices and dlm/clvmd devices from the cluster and
removed -W from /etc/sysconfig/sbd. Both nodes are still rebooted after a
short while after starting cluster with 'pcs cluster start --all'.

----

virt-050# grep -v \# /etc/sysconfig/sbd' | sort | uniq
SBD_DELAY_START=no
SBD_OPTS=
SBD_PACEMAKER=yes
SBD_STARTMODE=clean
SBD_WATCHDOG_DEV=/dev/watchdog
SBD_WATCHDOG_TIMEOUT=5

virt-050# pcs config
Cluster Name: STSRHTS16341
Corosync Nodes:
 virt-050 virt-051 
Pacemaker Nodes:
 virt-050 virt-051 

Resources: 

Stonith Devices: 
Fencing Levels: 

Location Constraints:
Ordering Constraints:
Colocation Constraints:

Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: STSRHTS16341
 dc-version: 1.1.12-a14efad
 have-watchdog: false
 no-quorum-policy: freeze
 stonith-watchdog-timeout: 0s

[root@virt-050 ~]# systemctl status sbd
sbd.service - Shared-storage based fencing daemon
   Loaded: loaded (/usr/lib/systemd/system/sbd.service; disabled)
   Active: inactive (dead)

[root@virt-050 ~]# pcs cluster stop --all --wait
virt-051: Stopping Cluster (pacemaker)...
virt-050: Stopping Cluster (pacemaker)...
virt-050: Stopping Cluster (corosync)...
virt-051: Stopping Cluster (corosync)...

[root@virt-050 ~]# rpm -q corosync pacemaker
corosync-2.3.4-5.el7.x86_64
pacemaker-1.1.12-22.el7_1.2.x86_64
[root@virt-050 ~]# pcs cluster status
Error: cluster is not currently running on this node

[root@virt-050 ~]# systemctl status corosync
corosync.service - Corosync Cluster Engine
   Loaded: loaded (/usr/lib/systemd/system/corosync.service; disabled)
   Active: inactive (dead)

...
[root@virt-050 ~]# systemctl enable sbd
ln -s '/usr/lib/systemd/system/sbd.service' '/etc/systemd/system/corosync.service.requires/sbd.service'
[root@virt-050 ~]# ssh virt-051 systemctl enable sbd
Warning: Permanently added 'virt-051,10.34.71.51' (ECDSA) to the list of known hosts.
ln -s '/usr/lib/systemd/system/sbd.service' '/etc/systemd/system/corosync.service.requires/sbd.service'

[root@virt-050 ~]# pcs cluster start --all --wait
virt-051: Starting Cluster...
virt-050: Starting Cluster...

[root@virt-050 ~]# pcs config
Cluster Name: STSRHTS16341
Corosync Nodes:
 virt-050 virt-051 
Pacemaker Nodes:
 virt-050 virt-051 

Resources: 

Stonith Devices: 
Fencing Levels: 

Location Constraints:
Ordering Constraints:
Colocation Constraints:

Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: STSRHTS16341
 dc-version: 1.1.12-a14efad
 have-watchdog: true
 no-quorum-policy: freeze
 stonith-watchdog-timeout: 0s

[root@virt-050 ~]# pcs status
Cluster name: STSRHTS16341
WARNING: no stonith devices and stonith-enabled is not false
Last updated: Tue Jun  2 16:32:34 2015
Last change: Tue Jun  2 16:29:19 2015
Stack: corosync
Current DC: virt-051 (2) - partition with quorum
Version: 1.1.12-a14efad
2 Nodes configured
0 Resources configured

Online: [ virt-050 virt-051 ]

Full list of resources:

PCSD Status:
  virt-050: Online
  virt-051: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

<reboot occurs after ~1s>

Does /dev/watchdog exist?

It seems you have:

        <nvpair id="cib-bootstrap-options-stonith-watchdog-timeout" name="stonith-watchdog-timeout" value="0s"/>

Fairly sure that wouldn't be helping.

I cannot reproduce this, even with stonith-watchdog-timeout: 0s
There are also no logs from the nodes at the time of the reported symptoms.

Can I get access to the nodes?

By adding this to the sysconfig file: 

SBD_OPTS="-v -v"

and tailing /var/log/messages, I was able to see that we were hitting this log message:

        LOGONCE(pcmk_health_unknown, LOG_WARNING, "Node state: UNKNOWN");

Essentially there is a mis-match between what the cluster knows the node as, and what sbd knows it as.

Setting SBD_OPTS="-n virt-073" allowed the node to start normally.
I will change this message in a future version to be:

        LOGONCE(pcmk_health_unknown, LOG_WARNING, "Node state: %s is UNKNOWN", local_uname);

To keep this information for future use: the names pacemaker use for nodes
needs to match the names sbd uses.

To achieve that either use $(uname -n) for node names (which is sbd default) OR
use '-n <node name' parameter in /etc/sysconfig/sbd SBD_OPTS parameter to tell 
sbd which names you use for nodes.

I believe this can be closed as NOTABUG.

Ack. Closing