Bug 1550167

osp11

+++ This bug was initially created as a clone of Bug #1545666 +++

Description of problem:
I would like to report a possible fault in validation script:

/usr/share/openstack-tripleo-heat-templates/validation-scripts/all-nodes.sh

There is a function 

function ping_retry() {
  local IP_ADDR=$1
  local TIMES=${2:-'10'}
  local COUNT=0
  local PING_CMD=ping
  if [[ $IP_ADDR =~ ":" ]]; then
    PING_CMD=ping6
  fi
  until [ $COUNT -ge $TIMES ]; do
    if $PING_CMD -w 300 -c 1 $IP_ADDR &> /dev/null; then
      echo "Ping to $IP_ADDR succeeded."
      return 0
    fi
    echo "Ping to $IP_ADDR failed. Retrying..."
    COUNT=$(($COUNT + 1))
  done
  return 1
}

Problematic line here is

$PING_CMD -w 300 -c 1 $IP_ADDR

According to man ping:
       -w deadline
              Specify  a  timeout,  in seconds, before ping exits regardless of how many packets have been sent or received. In this case ping does not stop after count packet are sent, it waits either for deadline expire or until count probes are answered or for some error notification from network.

So, "-w 300" means 300 seconds deadline wait. and repeat this up to 10 times as per TIMES variable
This would provide a timeout of 3000 seconds or 50 minutes for the ping to complete on the worst case scenario.


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-5.3.3-1.el7ost.noarch.rpm



Actual results:
if "some error notification from network" is received, the ping will not wait "-w 300", it will exit quite fast.

Here is a test, where I ping a non-existing IP address in a existing subnet:

[VNF11 VPOD3 stack@director validation-scripts]$ time ping -w 300 -c 1 10.33.110.150
PING 10.33.110.150 (10.33.110.150) 56(84) bytes of data.
From 10.33.110.133 icmp_seq=1 Destination Host Unreachable
From 10.33.110.133 icmp_seq=2 Destination Host Unreachable
From 10.33.110.133 icmp_seq=3 Destination Host Unreachable
From 10.33.110.133 icmp_seq=4 Destination Host Unreachable

--- 10.33.110.150 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 2999ms
pipe 4

real    0m3.008s
user    0m0.000s
sys     0m0.002s

The ping exited in 3 seconds. If this happens during deployment, it would provide a timeout of 3 * 10 = 30 seconds.

30 seconds is too short if this ping must be made over bonded interface with LACP.



Expected results:

It takes anywhere between 30 and 60 seconds for LACP to become functional.

It does not matter is the slow or fast LACP mode is used on the switches, 30 seconds is a borderline minimum, and is not enough.

--- Additional comment from Dariusz Wojewódzki on 2018-02-15 08:33:13 EST ---

I noticed this has been changed in OSP12, openstack-tripleo-heat-templates-7.0.3-22.el7ost.noarch.rpm:


    if $PING_CMD -w 10 -c 1 $IP_ADDR &> /dev/null; then


Is it possible to implement it to OSP11 and OSP10?