Description of problem:

According to a user at AWS: "Once the fencing agent triggers an action (i.e.: StopInstance) it will run DescribeInstance with 1 second interval, which could cause Throttling and unexpected behaviors. It is not recommended to "burst" DescribeInstance like that and a check every 20s/30s would be sufficient in my view."

We're awaiting details from the customer about the type of behavior such throttling might cause and any potential impact to the (a) the cluster performing fencing or (b) any other clusters or instances in the environment.

This 1-second delay is hard-coded into async_set_multi_power_fn() in fencing.py. If this RFE is acked, we could either:
  (a) add the option for only the fence_aws agent, or
  (b) add the option for all fence agents.

With option (a), we could default it to either 1 (the current behavior) or to whatever we determine is a good balance between quick fencing and AWS best practices. The customer suggests that we "implement a waiter handler so it automatically waits until the instance switches to 'Stopped' without a need for a 'DescribeInstance' call." For example:
  - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Waiter.InstanceStopped

There is also a handler for InstanceRunning:
  - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Waiter.InstanceRunning


With option (b), we would want to default it to 1, to keep the existing quick-fencing behavior unless configured otherwise.

-----

Version-Release number of selected component (if applicable):

fence-agents-aws-4.2.1-30.el8_1.1.noarch
fence-agents-common-4.2.1-30.el8_1.1.noarch

-----

Steps to Test:

1. Configure the newly added delay/sleep_time option.
2. Use some form of debugging (e.g., print statements, verbose output, etc.) to verify how often the get_power_status() calls within the fence agent are being made.

-----

Additional info:

Customer is willing to assist in testing and provide info/suggestions from the AWS side.