Bug 1127752

Staypuft uses the controller_ips method in several places to template
class parameters, one of the most important being
pacemaker_cluster_members.  It is imperative that this method returns
a consistent list of IP addresses, because it directly drives the
pacemaker configuration file, and any change will break the cluster.

Presently, staypuft generates this list using the Host.ip value for
each host in the deployment.  This is from the Foreman Host model.
Foreman maintains the value of ip based on the value provided in the
ipaddress fact from facter.  In turn, facter takes the value of
ipaddress to be the first address encountered in the output of
ifconfig that does not match /^127./.

Here's what we've observed happening during an HA deployment:

- All three controllers provision on the management network, and run
  puppet during kickstart %post.  The ipaddress fact for each is the
  management network address, so this is correct so far.

- Hosts reboot, run puppet.  At this point it gets a little ambiguous.
  If the management interface is the first alphabetically, it will
  still be chosen as the ipaddress fact and the foreman host will
  still be correct.  However it's possible that the host provisioned
  off of eth1, and now the host has brought up eth0 as a DHCP
  interface.  In that case, ipaddress will be the eth0 address, and
  foreman will update the host ip accordingly.  This will cause
  controller_ips to change on the next puppet run.

- If we manage to get through the first puppet runs correctly across
  all controllers (e.g. eth0 was in fact the management interface),
  then the second time puppet runs the external bridge address will
  almost certainly take over the host ip, since br-ex comes before eth
  alphabetically.  The next puppet run will cause the controller_ips
  value to change to reflect this, and the pacemaker cluster will
  break.