Bug 1703128

(In reply to Klaus Wenninger from comment #0)

> 
> Additional info:
> 
> Fix of bz1702727 already leads to a reboot after watchdog-timeout.
> What we might like to see is that it is not possible to configure
> watchdog-timeout < quorum.device.sync_timeout.
> Or at least have documentation point us to not to do so.

As we chose to use a corosync-API (votequorum) that isn't stalled
while corosync is syncing for bz1702727 we won't get a reboot
after watchdog-timeout if corosync is stuck in syncing for longer
than watchdog-timeout.

Ensuring that watchdog-timeout is < quorum.device.sync_timeout is still desirable of course
to ensure that we don't get

- unwanted reboots due to watcher stalled via corosync-API
- we ensure on the other hand that quorum-loss triggered suicide
  will come early enough to prevent split-brain.

https://github.com/ClusterLabs/sbd/pull/114

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sbd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4822

Great. Another ridiculous problem in Red Hat's HA garbage stack.

Jan 06 12:04:21 audctstmr001.au.harveynorman.com sbd[3564258]:    cluster:  warning: set_servant_health: Connected to corosync but quorum using qdevice is distrusted for SBD as qdevice-sync_timeout (30s) > watchdog-timeout (5s).
Jan 06 12:05:51 audctstmr001.au.harveynorman.com systemd[1]: sbd.service: start operation timed out. Terminating.
Jan 06 12:05:51 audctstmr001.au.harveynorman.com sbd[3564256]:  warning: cleanup_servant_by_pid: Servant for pcmk (pid: 3564257) has terminated
Jan 06 12:05:51 audctstmr001.au.harveynorman.com sbd[3564256]:  warning: cleanup_servant_by_pid: Servant for cluster (pid: 3564258) has terminated
Jan 06 12:05:51 audctstmr001.au.harveynorman.com systemd[1]: sbd.service: Failed with result 'timeout'.
Jan 06 12:05:51 audctstmr001.au.harveynorman.com systemd[1]: Failed to start Shared-storage based fencing daemon.


So i can't start sbd. Why? I don't know.

This bug report is completely unintelligible & contradicts itself:

In comment 5 Klaus says: "Ensuring that watchdog-timeout is < quorum.device.sync_timeout is still desirable"
Eh? What? 
That is the opposite of what this errata is now enforcing.

If I do a googfle search for: qdevice-sync_timeout sbd
i find this bug report is about the only promising result.
Nothing else, anywhere, explains how qdevice-sync_timeout should be set when using quorum node with sdb.

Once again, Red Hat & Suse completely fail to document things properly.
Hopeless.

Ah ok, i have found this:
https://access.redhat.com/articles/2941601
Which says this:

"If you have configured your system to run a quorum device and the value of SBD_WATCHDOG_TIMEOUT is less than the value of qdevice-sync_timeout, a quorum state update could be delayed for so long that it would result in a split-brain situation. The value of qdevice-sync_timeout is not currently configurable by means of PCS and should be considered non-configurable with a default of 30s. Red Hat suggests a difference of 3-5 seconds between SBD_WATCHDOG_TIMEOUT and qdevice-sync_timeout, and that stonith-watchdog-timeout exceed SBD_WATCHDOG_TIMEOUT by double its value (or should be zero to indicate that watchdog fencing should not be used). As of RHEL 8.3, Pacemaker will not start up if the SBD_WATCHDOG_TIMEOUT value and the qdevice-sync_timeout value do not match, and if these mismatched parameters are configured on a running system SBD will issue a reboot."

So, if i try to make sense of all of this nonsense, the upshot is:
1) qdevice-sync_timeout is nonconfigurable, and stuck at 30s
2) SBD_WATCHDOG_TIMEOUT must be > qdevice-sync_timeout, and Red Hat suggests a difference of 3-5 seconds.
3) SBD_WATCHDOG_TIMEOUT should therefore be, say 34 seconds
4) "stonith-watchdog-timeout exceed SBD_WATCHDOG_TIMEOUT by double its value" - now, this is extremely poor english, which i believe is NOT what red hat actually intended.
To say "A exceeds B by double its value", in fact means that (A-B)=2B, ie A=3*B.

So, if I take this pathetic documentation at face-value, it is telling me stonith-watchdog-timeout should be 3 * 34 = 102 seconds.
Of course, I do not believe "A exceeds B by double its value" was intended to mean A=3*B.
I think it was intended to mean A=2*B, in which case we have stonith-watchdog-timeout=68 seconds.

Is the upshot of all this, that in a 2-NODE cluster with quorum device as 3rd node, then we have:
SBD_WATCHDOG_TIMEOUT=34
stonith-watchdog-timeout=68

That seems to be waaay off the normal values for those parameters, which is SBD_WATCHDOG_TIMEOUT=5, & stonith-watchdog-timeout=10.

Thanks very much Red Hat, for your completely misleading and useless documentation.

Can anyone actually confirm what this garbage documentation really means?

(In reply to John from comment #30)
> Great. Another ridiculous problem in Red Hat's HA garbage stack.
> 
> Jan 06 12:04:21 audctstmr001.au.harveynorman.com sbd[3564258]:    cluster: 
> warning: set_servant_health: Connected to corosync but quorum using qdevice
> is distrusted for SBD as qdevice-sync_timeout (30s) > watchdog-timeout (5s).
> Jan 06 12:05:51 audctstmr001.au.harveynorman.com systemd[1]: sbd.service:
> start operation timed out. Terminating.
> Jan 06 12:05:51 audctstmr001.au.harveynorman.com sbd[3564256]:  warning:
> cleanup_servant_by_pid: Servant for pcmk (pid: 3564257) has terminated
> Jan 06 12:05:51 audctstmr001.au.harveynorman.com sbd[3564256]:  warning:
> cleanup_servant_by_pid: Servant for cluster (pid: 3564258) has terminated
> Jan 06 12:05:51 audctstmr001.au.harveynorman.com systemd[1]: sbd.service:
> Failed with result 'timeout'.
> Jan 06 12:05:51 audctstmr001.au.harveynorman.com systemd[1]: Failed to start
> Shared-storage based fencing daemon.
> 
> 
> So i can't start sbd. Why? I don't know.

If sbd starts up, pacemaker will do as well and it will start
resources - with split-brain-risk.
As to prevent split-brain-risk sbd won't start an issue the warning above
(admittedly should be an error message rather) that does exactly
tell what is wrong.

> 
> This bug report is completely unintelligible & contradicts itself:
> 
> In comment 5 Klaus says: "Ensuring that watchdog-timeout is <
> quorum.device.sync_timeout is still desirable"
> Eh? What? 
> That is the opposite of what this errata is now enforcing.
> 

Obviously wrong way round but that is not user-documentation and most
people (including me) that dealt with this bz didn't even notice that
because it is so obvious.

In general what you are seeing is the first step in a 2 step
approach:

- prevent undesirable operation by staying down if configuration
  is inconsistent

- make high-level-tooling (pcs, crmsh, ...) prevent setting
  inconsistent values

I btw. consider clusterlabs mailing list a better place for
discussions like this instead of commenting on closed bugs.

Regards,
Klaus