Bug 1389413

Description of problem:

Per bz#1211781, we went through a lot of testing to come up with the best settings for HAProxy on Galera which include that the "inter" setting is one second, with no retry of any kind.  Because this is Galera and we are using a simple inetd-based health checker, if this check returns "down", failover needs to happen immediately otherwise openstack services will begin to error out, rather than being able to reconnect and get onto a good node.

Those settings look like:

listen galera
  bind 192.168.0.13:3306
  mode  tcp
  option  tcplog
  option  httpchk
  option  tcpka
  stick  on dst
  stick-table  type ip size 1000
  timeout  client 90m
  timeout  server 90m
  server pcmk-maca25400702876 192.168.0.7:3306  check inter 1s port 9200 backup on-marked-down shutdown-sessions
  server pcmk-maca25400702877 192.168.0.10:3306  check inter 1s port 9200 backup on-marked-down shutdown-sessions
  server pcmk-maca25400702875 192.168.0.9:3306  check inter 1s port 9200 backup on-marked-down shutdown-sessions


That BZ affected the foreman installer, and it appears that in the current tripleo installer, these settings have been lost, now we are getting:

listen mysql
  bind ip:3306 
  option httpchk
  stick on dst
  stick-table type ip size 1000
  timeout client 90m
  timeout server 90m
  server overcloud-controller-0 ip:3306 backup check fall 5 inter 2000 on-marked-down shutdown-sessions port 9200 rise 2
  server overcloud-controller-1 ip:3306 backup check fall 5 inter 2000 on-marked-down shutdown-sessions port 9200 rise 2
  server overcloud-controller-2 ip:3306 backup check fall 5 inter 2000 on-marked-down shutdown-sessions port 9200 rise 2

Above, the health check is every two seconds, not one, and the health check needs to run five times before it determines the node is down.  This means it takes ten full seconds of a Galera node that's down and unresponsive to be taken out of the proxy, during which time all openstack services will be non-functioning and will return errors to end users.  This has led to bz#1381485 where openstack services are erroring out unnecessarily and then has caused lots of confusion on the customer's part as to how to resolve.

The source to this can be seen via the defaults at https://github.com/redhat-openstack/openstack-puppet-modules/blob/stable/mitaka/tripleo/manifests/loadbalancer.pp#L384 leading to https://github.com/redhat-openstack/openstack-puppet-modules/blob/stable/mitaka/tripleo/manifests/loadbalancer.pp#L1338 where the "inter 1s" is not being set, and the "fall 5 inter 2000" is being kept.


Version-Release number of selected component (if applicable):

All tripleo

How reproducible:

always

Steps to Reproduce:
1. run tripleo installer
2. haproxy.cfg has wrong values

Actual results:

openstack services have a ten second delay while a galera node is offline before they can recover

Expected results:

there is at most a one second delay while a galera node is offline, allowing most openstack applications to recover without errors.