Two general responses to this come to mind:

The first, and obviously simplest, is to change nothing, document the issue in the KB, and recommend that users set the pcmk_reboot_timeout to a value more than (token) + (consensus) + (how long they expect the reboot to take). The only downside I see for this -- besides that the issue would ideally be handled automatically -- is that most users and support engineers currently believe pcmk_reboot_timeout only needs to be set to a value higher than "how long they expect the reboot to take". It's never an exact science anyway and usually gets increased in increments of 60 to 240 seconds at a time, so IMO this wouldn't be an undue hardship.

The second is to avoid initiating a second fence attempt but ensure the reply is received before declaring the action complete. Something like "if we have an operation result locally (i.e., the async command completed) within the stonith action timeout, then keep waiting for the async reply to be delivered but don't mark the action as timed out." There may or may not be a safe way to implement that.


I read a little bit more about CPG and virtual synchrony last night, and it sounds like for better or for worse we're stuck waiting for that reply and won't get it until the fenced node leaves the membership.

(In reply to Reid Wahl from comment #1)
> The second is to avoid initiating a second fence attempt but ensure the
> reply is received before declaring the action complete. Something like "if
> we have an operation result locally (i.e., the async command completed)
> within the stonith action timeout, then keep waiting for the async reply to
> be delivered but don't mark the action as timed out." There may or may not
> be a safe way to implement that.

D'oh. The origin may not be same node that's executing fencing, if the action was delegated. That renders #2 non-feasible as described.

Two-node test cluster tunnel vision :)

One other idea occurred to me: add the token timeout and consensus timeout to the stonith action timeout (#if SUPPORT_COROSYNC). Maybe we could do this conditionally, only if the node-to-be-fenced is still part of the CPG membership when fencing is initiated. There's no reason to add the extra timeout if the node has already left the membership and is getting fenced due to token loss.

This way, only the timeout would change. We wouldn't be adding complexity or introducing bugs elsewhere.

If I follow, the issue is that the originator of fencing must wait for a CPG reply from the executor of fencing (even if it's the same node), and the CPG reply can't arrive until corosync detects the fenced node as having left the CPG. That means the token timeout eats into the fence action timeout, even if the fence action actually completed early in that period. If the fence action timeout is lower than the token timeout, it will always time out.

My first instinct is that the originator should wait the maximum of the fence action timeout and the corosync CPG timeout before declaring the operation timed out. The fencer would need to know the CPG timeout, possibly requiring a CFG API connection, and I'm not completely sure that's a good idea. We could maybe use the cluster-delay property instead, though we'd have to recommend that users set that above the token timeout (which may already be a practical requirement, I'm not sure).

Aside: there are no specific definitions of task, action, and operation in Pacemaker, and they are often used interchangeably. I lean toward using "action" for the generic type of action (start, reboot, etc.) and "operation" for a specific instance of that action, but it's inconsistent currently.

(In reply to Ken Gaillot from comment #4)
> If I follow, the issue is that the originator of fencing must wait for a CPG
> reply from the executor of fencing (even if it's the same node), and the CPG
> reply can't arrive until corosync detects the fenced node as having left the
> CPG. That means the token timeout eats into the fence action timeout, even
> if the fence action actually completed early in that period. If the fence
> action timeout is lower than the token timeout, it will always time out.

That sounds accurate. For fencing due to token loss, this is not an issue as long as the failed node doesn't rejoin the CPG membership before fencing completes. It can occur when a node is fenced for other reasons.


> My first instinct is that the originator should wait the maximum of the
> fence action timeout and the corosync CPG timeout before declaring the
> operation timed out. The fencer would need to know the CPG timeout, possibly
> requiring a CFG API connection, and I'm not completely sure that's a good
> idea. We could maybe use the cluster-delay property instead, though we'd
> have to recommend that users set that above the token timeout

Using cluster-delay is worth testing. I can't remember off the top of my head whether the timeout here includes the cluster-delay or not.

Setting pcmk_<action>_timeout is already a valid workaround, although it requires accounting for both the token timeout and the time it takes for the stonith action to complete.


> (which may
> already be a practical requirement, I'm not sure).

Not that I know of. We have customers setting the token timeout to values as high as 300000 (the max supported value), and I can't recall ever seeing a customer override the default cluster-delay. It hasn't caused any issues that we've identified yet (besides potentially this one).

(In reply to Reid Wahl from comment #5)
> (In reply to Ken Gaillot from comment #4)
> > If I follow, the issue is that the originator of fencing must wait for a CPG
> > reply from the executor of fencing (even if it's the same node), and the CPG
> > reply can't arrive until corosync detects the fenced node as having left the
> > CPG. That means the token timeout eats into the fence action timeout, even
> > if the fence action actually completed early in that period. If the fence
> > action timeout is lower than the token timeout, it will always time out.
> 
> That sounds accurate. For fencing due to token loss, this is not an issue as
> long as the failed node doesn't rejoin the CPG membership before fencing
> completes. It can occur when a node is fenced for other reasons.

That makes sense. It sounds like the possibility is fairly small, with the most likely scenario being a node gradually becoming unusable (due to load or network issues, etc.), something like a stop failure being the first fenceable symptom, and the node dropping out completely just after the fence action is initiated.

> > My first instinct is that the originator should wait the maximum of the
> > fence action timeout and the corosync CPG timeout before declaring the
> > operation timed out. The fencer would need to know the CPG timeout, possibly
> > requiring a CFG API connection, and I'm not completely sure that's a good
> > idea. We could maybe use the cluster-delay property instead, though we'd
> > have to recommend that users set that above the token timeout
> 
> Using cluster-delay is worth testing. I can't remember off the top of my
> head whether the timeout here includes the cluster-delay or not.
> 
> Setting pcmk_<action>_timeout is already a valid workaround, although it
> requires accounting for both the token timeout and the time it takes for the
> stonith action to complete.
> 
> 
> > (which may
> > already be a practical requirement, I'm not sure).
> 
> Not that I know of. We have customers setting the token timeout to values as
> high as 300000 (the max supported value), and I can't recall ever seeing a

:-o

> customer override the default cluster-delay. It hasn't caused any issues
> that we've identified yet (besides potentially this one).

In that case it's probably best not to overload cluster-delay.

A solution that occurs to me would be to track the token timeout (via a CFG connection) in the fencer, and add it to the fence action timeout when the target is an active cluster member.

Given the rarity, the limited impact, and the availability of a workaround, I'm setting this low priority. For now, we can document the issue.

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.