Bug 1774143

Most customers accept the default totem token setting in their clusters.  However, it is very common for our customers to experience node fencing as a result of temporary system unresponsiveness, and usually we have to suggest raising totem token before a customer will even consider that as a possible adjustment.  

Also, many admins don't even know what totem token is, and so they're not finding the documentation or options on their own if they want to dial-back how aggressive their cluster is in fencing nodes.

The Red Hat Support team has long dealt with a high volume of customers asking for root-cause analysis of fence events or node-reboots.  The most common scenario for node fencing is a node becoming temporarily unresponsive longer than corosync's communication timeout, and many customers are left with the impression that RHEL HA is "unstable" when they see node fencing, when in reality a 1s timeout is often just too aggressive for many environments. 

Even though you can set token timeout at setup-time with 'pcs cluster setup totem ...', there is no way yet to do this after the fact.  Also, like I said, many customers don't know what totem is, or what token is, so even with the setup option many admins miss this.

The goal of this request is to make it easier for customers to raise the totem token value using pcs.  I would like to propose a few different ways to do that:

1) Offer a command to change totem token after setup, such as: 

   # pcs cluster update --token=10s


2) Make a command to do the same but with more obvious terminology and simpler for a novice administrator to figure out.  Maybe we could have variations of this under both 'pcs cluster' and 'pcs stonith'.   For example: 

   # pcs cluster communication-timeout (--aggressive|--moderate|--lenient|--timeout=X)

   # pcs stonith communication-timeout (--aggressive|--moderate|--lenient|--timeout=X)

and those options might set something like 1s, 10s, 60s.  


3) In bug #1774132, I proposed we add a 'pcs diagnostics' command, and if we do that, we could offer a variation of this token-adjustment as a subcommand.  For example:

   # pcs diagnostics communication-timeout (--aggressive|--moderate|--lenient|--timeout=X)

   # pcs diagnostics setup  # (which would deploy all the diagnostics, including a --moderate communication timeout).


Obviously these are all redundant.  My goal with these suggestions is to make it more obvious to admins that they can tweak how aggressive the cluster is with fencing, and to put those commands in several places so they'll be more likely to actually discover it. 

My top priority is to at least make it possible to update token through SOME command, so we don't have to guide customers through a config file adjustment whenever there is a fencing incident.  

Once we can do that, then these other suggestions seem like they'd be easy to add as alternative ways to do the same thing.