Bug 1329756

Summary: Predictable IPs conflict with VIPs
Product: Red Hat OpenStack Reporter: Jon Jozwiak <jjozwiak>
Component: openstack-tripleo-heat-templatesAssignee: Emilien Macchi <emacchi>
Status: CLOSED WONTFIX QA Contact: Arik Chernetsky <achernet>
Severity: medium Docs Contact:
Priority: medium    
Version: 10.0 (Newton)CC: bnemec, curtis, gcharot, jraju, kholden, mburns, mcornea, mgarciam, nmorell, rcernin, rhel-osp-director-maint, srevivo
Target Milestone: ---Keywords: Documentation, Triaged, ZStream
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-19 22:33:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Jon Jozwiak 2016-04-22 21:07:23 UTC 
Description of problem:
Documentation states how to configure predictable IPs here: 
https://access.redhat.com/documentation/en/red-hat-openstack-platform/version-8/director-installation-and-usage/#sect-Predictable_IPs

However, if the IPs you pick are at the start of your usable range, the deployment will fail because the VIPs will automatically take the first IPs even if they are specified for use in the ips-from-pool-all.yaml.  This results in things like galera or redis not starting up properly in the deployment but otherwise doesn't show the conflict.  


Version-Release number of selected component (if applicable):
RHEL OSP 8 GA

How reproducible:
It is possible to reproduce every time your IPs in ips-from-pool-all.yaml are at the start of your IP range.  

Steps to Reproduce:
1. Create a network-isolation.yaml file with allocation pools like this: 
  InternalApiNetCidr: 192.168.221.0/24
  InternalApiAllocationPools: [{'start': '192.168.221.20', 'end': '192.168.221.200'}]
  #InternalApiNetworkVlanID: ""

2. Create an ips-from-pool-all.yaml file with hosts starting at the beginning of the pool.  For example: 
  ControllerIPs:
    # Each controller will get an IP from the lists below, first controller, first IP
    internal_api:
    - 192.168.221.20
    - 192.168.221.21
    - 192.168.221.22

3.  Deploy OpenStack with these files 
openstack overcloud deploy --templates ~/templates/ --ntp-server 0.fedora.pool.ntp.org --libvirt-type kvm --control-flavor control --compute-flavor compute --ceph-storage-flavor ceph-storage --control-scale 3 --compute-scale 2 --ceph-storage-scale 3 --neutron-tunnel-types vxlan --neutron-network-type vxlan -e ~/templates/environments/storage-environment.yaml -e ~/templates/environments/network-isolation.yaml -e ~/templates/environments/ips-from-pool-all.yaml

Actual results:
The deploy will set a VIP to 192.168.221.20, but a controller will also have the IP 192.168.221.20.  This results in some services failing and strange behavior due to the IP conflict

Expected results:
Ideally our documentation would call out this risk and point out how to avoid this.  

Additional info:
We should update our documentation to include a warning to NOT set the hosts in the ips-from-pool-all.yaml file to the beginning of the range.  This will avoid IP conflicts.  

In addition, we should include the following to also predictably assign VIPs: 

To predictably assign VIPs, update your network isolation file to include the required details: 

vi ~/templates/environments/network-isolation.yaml

# In the resource_registry section update the port assignment values as follows: 

 # Port assignments for the VIPs
  OS::TripleO::Network::Ports::NetVipMap: ../network/ports/net_vip_map_external.yaml
  OS::TripleO::Network::Ports::ExternalVipPort: ../network/ports/noop.yaml
  OS::TripleO::Network::Ports::InternalApiVipPort: ../network/ports/noop.yaml
  OS::TripleO::Network::Ports::StorageVipPort: ../network/ports/noop.yaml
  OS::TripleO::Network::Ports::StorageMgmtVipPort: ../network/ports/noop.yaml
  OS::TripleO::Network::Ports::RedisVipPort: ../network/ports/from_service.yaml


# In the parameter_defaults section add VIPs for each network as follows (with IPs updated to align with your network):

  ControlPlaneIP: 192.168.200.100
  ExternalNetworkVip: 192.168.225.100
  InternalApiNetworkVip: 192.168.221.100
  StorageNetworkVip: 192.168.223.100
  StorageMgmtNetworkVip: 192.168.224.100
  ServiceVips:
    redis: 192.168.221.101
Comment 2 Marcos Garcia 2017-01-26 02:25:23 UTC 
I can confirm I have the same behavior in OSP10. Deployment fails with VIPs set in the beginning of the pool

The VIP is the same as a regular IP assigned form the Pool to nodes, i.e. a storage node has the same IP as the VIP of the controller for the StorageNetworkVIP

I am redeploying with an IP in the middle of the range, to see if this is an edge case.

It's a bug, nonetheless, and a workaround could be described in the documentation page https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/single/advanced-overcloud-customization/#sect-Predictable_VIPs


NOTE: once the deployment have failed, I inspect all nodes for all their IPs with the following command (first do source stackrc)

# 
ansible overcloud -i /bin/tripleo-ansible-inventory  -u heat-admin --ssh-extra-args='-o StrictHostKeyChecking=no' -m shell -a "ip a | grep inet\ “ > output.txt

Then look for duplicate IPs in output.txt
[stack@undercloud ~]$ sort output.txt  |grep global
    inet 10.12.48.100/32 brd 10.12.48.255 scope global br-ex
    inet 10.12.48.105/24 brd 10.12.48.255 scope global br-ex
    inet 10.12.48.109/24 brd 10.12.48.255 scope global br-ex
    inet 10.12.48.112/24 brd 10.12.48.255 scope global br-ex
    inet 172.16.3.20/24 brd 172.16.3.255 scope global br-mgmt
    inet 172.16.3.20/32 brd 172.16.3.255 scope global br-mgmt
    inet 172.16.3.23/24 brd 172.16.3.255 scope global br-mgmt
    inet 172.16.3.25/32 brd 172.16.3.255 scope global br-mgmt
    inet 172.16.3.26/24 brd 172.16.3.255 scope global br-mgmt
    inet 172.16.3.27/24 brd 172.16.3.255 scope global br-mgmt
    inet 172.16.3.28/24 brd 172.16.3.255 scope global br-mgmt
    inet 172.16.3.31/24 brd 172.16.3.255 scope global br-mgmt
    inet 172.16.3.37/24 brd 172.16.3.255 scope global br-mgmt
    inet 172.16.4.20/24 brd 172.16.4.255 scope global enp8s0
    inet 172.16.4.20/32 brd 172.16.4.255 scope global enp8s0
    inet 172.16.4.21/24 brd 172.16.4.255 scope global enp8s0
    inet 172.16.4.24/24 brd 172.16.4.255 scope global enp8s0
    inet 172.16.4.27/24 brd 172.16.4.255 scope global enp8s0
    inet 172.16.4.29/24 brd 172.16.4.255 scope global enp8s0
    inet 172.16.4.30/24 brd 172.16.4.255 scope global enp8s0
    inet 172.16.4.38/24 brd 172.16.4.255 scope global enp8s0
    inet 172.16.5.20/32 brd 172.16.5.255 scope global enp13s0
    inet 172.16.5.28/24 brd 172.16.5.255 scope global enp13s0
    inet 172.16.5.33/24 brd 172.16.5.255 scope global enp13s0
    inet 172.16.5.35/24 brd 172.16.5.255 scope global enp13s0
    inet 172.16.6.20/24 brd 172.16.6.255 scope global enp15s0
    inet 172.16.6.23/24 brd 172.16.6.255 scope global enp15s0
    inet 172.16.6.25/24 brd 172.16.6.255 scope global enp15s0
    inet 172.16.6.26/24 brd 172.16.6.255 scope global enp14s0
    inet 172.16.6.29/24 brd 172.16.6.255 scope global enp14s0
    inet 172.16.6.30/24 brd 172.16.6.255 scope global enp14s0
    inet 172.16.6.32/24 brd 172.16.6.255 scope global enp14s0
    inet 192.168.0.21/24 brd 192.168.0.255 scope global enp6s0
    inet 192.168.0.2/32 brd 192.168.0.255 scope global enp6s0
    inet 192.168.0.25/24 brd 192.168.0.255 scope global enp6s0
    inet 192.168.0.28/24 brd 192.168.0.255 scope global enp6s0
    inet 192.168.0.31/24 brd 192.168.0.255 scope global enp6s0
    inet 192.168.0.32/24 brd 192.168.0.255 scope global enp6s0
    inet 192.168.0.34/24 brd 192.168.0.255 scope global enp6s0
    inet 192.168.0.37/24 brd 192.168.0.255 scope global enp6s0

As you can see, some of the IPs ending in .20 appear in /24 (regular IP of a NIC) and /32 (VIP in a controlleR) - like 172.16.4.20 and 172.16.3.20

[stack@undercloud ~]$ tail templates/network-config.yaml  -n 15
  # Customize bonding options, e.g. "mode=4 lacp_rate=1 updelay=1000 miimon=100"
  BondInterfaceOvsOptions: "mode=active-backup"
  ExternalNetworkVip: 10.12.48.100
  ControlPlaneIP: 192.168.0.2
  InternalApiNetworkVip: 172.16.3.20
  StorageNetworkVip: 172.16.4.20
  StorageMgmtNetworkVip: 172.16.5.20
  ServiceVips:
    redis: 172.16.3.21
Comment 3 Gregory Charot 2017-01-26 10:29:48 UTC 
The documentation states :

Predictable IPs :

"Make sure the chosen IP addresses fall outside the allocation pools for each network defined in your network environment file”  [1]

Predictable VIPs :

"Select these IPs from their respective allocation pool ranges. For example, select the InternalApiNetworkVip from the InternalApiAllocationPools range.(...) This avoids any conflicts with the VIPs chosen for each network." [2]


I do confirm setting predictable IPs from the allocation pools ends up in a conflict nevertheless the documentation looks fine to me.

[1]https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/single/advanced-overcloud-customization/#sect-Predictable_IPs

[2]https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/single/advanced-overcloud-customization/#sect-Predictable_VIPs
Comment 4 Marcos Garcia 2017-01-26 21:16:37 UTC 
Confirmed that using Predictable VIPs that fall on the first part of the allocation block results in a failed deployment. I retried with a PredictableVIP at the end of the block, and it worked OK

It makes me think that PredictableVIPs do not play nice with regular dinamic IPs. As you can see, the VIPs I want are not in neutron port-list (undercloud)
[stack@undercloud ~]$ neutron port-list | grep virtual_ip
| 17d4be25-a921-42e4-9da5-3912525110a0 | internal_api_virtual_ip       | fa:16:3e:c1:a7:6c | {"subnet_id": "856ad1cb-38be-4253-a7d4-b2a73260999c", "ip_address": "172.16.3.27"}  |
| 3f7461bc-8349-4e65-a457-1495416441c5 | storage_management_virtual_ip | fa:16:3e:de:e0:c7 | {"subnet_id": "8899d7e6-099e-475b-b348-197002d18266", "ip_address": "172.16.5.21"}  |
| 47eb4883-0dfa-4c2f-94c1-7a5258a2ed0e | storage_virtual_ip            | fa:16:3e:54:92:e7 | {"subnet_id": "4b3bcc4f-25e1-4907-91b1-05660b7f211b", "ip_address": "172.16.4.20"}  |
| 534a9d3d-18a4-440b-9bab-cb10c8a190a6 | public_virtual_ip             | fa:16:3e:ee:bc:fc | {"subnet_id": "a83fb401-3036-4e9f-b233-7c522550c5e0", "ip_address": "10.12.48.106"} |
| bec0047a-eaea-4971-b12e-a062821503e4 | control_virtual_ip            | fa:16:3e:ac:76:5f | {"subnet_id": "2afc404d-94d2-4479-bbef-ee38e1bb0a9a", "ip_address": "192.168.0.26"} |
| c5b8be1c-bacb-4f90-9748-490fe8a0fda2 | redis_virtual_ip              | fa:16:3e:7c:72:66 | {"subnet_id": "856ad1cb-38be-4253-a7d4-b2a73260999c", "ip_address": "172.16.3.20"}  |

[stack@undercloud ~]$ cat templates/network-config.yaml 
#This file is an example of an environment file for defining the isolated
#networks and related parameters.
resource_registry:
  # Network Interface templates to use (these files must exist)
  OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute.yaml
  OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml
  OS::TripleO::CephStorage::Net::SoftwareConfig: /home/stack/templates/nic-configs/ceph-storage.yaml
  OS::TripleO::Network::Ports::NetVipMap: /usr/share/openstack-tripleo-heat-templates/network/ports/net_vip_map_external.yaml
  OS::TripleO::Network::Ports::ExternalVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::InternalApiVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::StorageVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::StorageMgmtVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/from_service.yaml

parameter_defaults:
  InternalApiNetCidr: 172.16.3.0/24
  StorageNetCidr: 172.16.4.0/24
  StorageMgmtNetCidr: 172.16.5.0/24
  TenantNetCidr: 172.16.6.0/24
  ExternalNetCidr: 10.12.48.0/24
  # CIDR subnet mask length for provisioning network
  ControlPlaneSubnetCidr: '24'
  # Customize the IP ranges on each network to use for static IPs and VIPs
  InternalApiAllocationPools: [{'start': '172.16.3.20', 'end': '172.16.3.100'}]
  StorageAllocationPools: [{'start': '172.16.4.20', 'end': '172.16.4.100'}]
  StorageMgmtAllocationPools: [{'start': '172.16.5.20', 'end': '172.16.5.100'}]
  TenantAllocationPools: [{'start': '172.16.6.20', 'end': '172.16.6.100'}]
  # Leave room if the external network is also used for floating IPs
  ExternalAllocationPools: [{'start': '10.12.48.100', 'end': '10.12.48.120'}]
  # Gateway router for the external network
  ExternalInterfaceDefaultRoute: 10.12.48.254
  # Gateway router for the provisioning network (or Undercloud IP)
  ControlPlaneDefaultRoute: 192.168.0.1
  # Generally the IP of the Undercloud
  EC2MetadataIp: 192.168.0.1
  # Define the DNS servers (maximum 2) for the overcloud nodes
  DnsServers: ["10.12.48.11","8.8.8.8"]
  # Customize the VLAN IDs to match the local environment
  InternalApiNetworkVlanID: 3 #NOTE: we do not use tags, all vNICs are in native mode
  StorageNetworkVlanID: 4 #NOTE: we do not use tags, all vNICs are in native mode
  StorageMgmtNetworkVlanID: 5 #NOTE: we do not use tags, all vNICs are in native mode
  TenantNetworkVlanID: 6 #NOTE: we do not use tags, all vNICs are in native mode
  # Set to empty string to enable multiple external networks or VLANs
  NeutronExternalNetworkBridge: "br-ex"
  # Customize bonding options, e.g. "mode=4 lacp_rate=1 updelay=1000 miimon=100"
  BondInterfaceOvsOptions: "mode=active-backup"
  ExternalNetworkVip: 10.12.48.100 #must be 100 so DNS can work
  ControlPlaneIP: 192.168.0.4
  InternalApiNetworkVip: 172.16.3.99
  StorageNetworkVip: 172.16.4.99
  StorageMgmtNetworkVip: 172.16.5.99
  ServiceVips:
    redis: 172.16.3.98
Comment 5 Marcos Garcia 2017-01-26 21:19:34 UTC 
My tests are not the same as Jon, as he tried BOTH predictableIP and predictableVIP, whereas I am only testing predictableVIPs, which does not work as documented.

(In reply to Gregory Charot from comment #3)
> The documentation states :
> 
> Predictable IPs :
> 
> "Make sure the chosen IP addresses fall outside the allocation pools for
> each network defined in your network environment file”  [1]
> 
> Predictable VIPs :
> 
> "Select these IPs from their respective allocation pool ranges. For example,
> select the InternalApiNetworkVip from the InternalApiAllocationPools
> range.(...) This avoids any conflicts with the VIPs chosen for each
> network." [2]
> 
> 
> I do confirm setting predictable IPs from the allocation pools ends up in a
> conflict nevertheless the documentation looks fine to me.
> 
> [1]https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/
> single/advanced-overcloud-customization/#sect-Predictable_IPs
> 
> [2]https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/
> single/advanced-overcloud-customization/#sect-Predictable_VIPs
Comment 6 Marcos Garcia 2017-01-26 23:45:25 UTC 
Testing a PredictableVIP inside the allocation pool, 2nd place to be exact (  InternalApiNetworkVip: 172.16.3.21 , where   InternalApiAllocationPools: [{'start': '172.16.3.20', 'end': '172.16.3.100'}]), makes the deployment fail here:

[stack@undercloud ~]$ openstack stack failures list overcloud
overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 696c261b-0ddf-45f7-b92e-ab960d3047eb
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    Matching apachectl 'Server version: Apache/2.4.6 (Red Hat Enterprise Linux)
    Server built:   Aug  3 2016 08:33:27'
    Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.
  deploy_stderr: |
    exception: connect failed
    Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
    Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
    Error: Duplicate declaration: Pcmk_resource[ip-172.16.3.21] is already declared in file /etc/puppet/modules/pacemaker/manifests/resource/ip.pp:106; cannot redeclare at /etc/puppet/modules/pacemaker/manifests/resource/ip.pp:106 on node overcloud-controller-0.localdomain
    Error: Duplicate declaration: Pcmk_resource[ip-172.16.3.21] is already declared in file /etc/puppet/modules/pacemaker/manifests/resource/ip.pp:106; cannot redeclare at /etc/puppet/modules/pacemaker/manifests/resource/ip.pp:106 on node overcloud-controller-0.localdomain


[stack@undercloud ~]$ cat templates/network-config.yaml 
#This file is an example of an environment file for defining the isolated
#networks and related parameters.
resource_registry:
  # Network Interface templates to use (these files must exist)
  OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute.yaml
  OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml
  OS::TripleO::CephStorage::Net::SoftwareConfig: /home/stack/templates/nic-configs/ceph-storage.yaml
  OS::TripleO::Network::Ports::NetVipMap: /usr/share/openstack-tripleo-heat-templates/network/ports/net_vip_map_external.yaml
  OS::TripleO::Network::Ports::ExternalVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::InternalApiVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::StorageVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::StorageMgmtVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/from_service.yaml

parameter_defaults:
  InternalApiNetCidr: 172.16.3.0/24
  StorageNetCidr: 172.16.4.0/24
  StorageMgmtNetCidr: 172.16.5.0/24
  TenantNetCidr: 172.16.6.0/24
  ExternalNetCidr: 10.12.48.0/24
  # CIDR subnet mask length for provisioning network
  ControlPlaneSubnetCidr: '24'
  # Customize the IP ranges on each network to use for static IPs and VIPs
  InternalApiAllocationPools: [{'start': '172.16.3.20', 'end': '172.16.3.100'}]
  StorageAllocationPools: [{'start': '172.16.4.20', 'end': '172.16.4.100'}]
  StorageMgmtAllocationPools: [{'start': '172.16.5.20', 'end': '172.16.5.100'}]
  TenantAllocationPools: [{'start': '172.16.6.20', 'end': '172.16.6.100'}]
  # Leave room if the external network is also used for floating IPs
  ExternalAllocationPools: [{'start': '10.12.48.100', 'end': '10.12.48.120'}]
  # Gateway router for the external network
  ExternalInterfaceDefaultRoute: 10.12.48.254
  # Gateway router for the provisioning network (or Undercloud IP)
  ControlPlaneDefaultRoute: 192.168.0.1
  # Generally the IP of the Undercloud
  EC2MetadataIp: 192.168.0.1
  # Define the DNS servers (maximum 2) for the overcloud nodes
  DnsServers: ["10.12.48.11","8.8.8.8"]
  # Customize the VLAN IDs to match the local environment
  InternalApiNetworkVlanID: 3 #NOTE: we do not use tags, all vNICs are in native mode
  StorageNetworkVlanID: 4 #NOTE: we do not use tags, all vNICs are in native mode
  StorageMgmtNetworkVlanID: 5 #NOTE: we do not use tags, all vNICs are in native mode
  TenantNetworkVlanID: 6 #NOTE: we do not use tags, all vNICs are in native mode
  # Set to empty string to enable multiple external networks or VLANs
  NeutronExternalNetworkBridge: "br-ex"
  # Customize bonding options, e.g. "mode=4 lacp_rate=1 updelay=1000 miimon=100"
  BondInterfaceOvsOptions: "mode=active-backup"
  ExternalNetworkVip: 10.12.48.100 #must be 100 so DNS can work
  ControlPlaneIP: 192.168.0.4
  InternalApiNetworkVip: 172.16.3.21
  StorageNetworkVip: 172.16.4.21
  StorageMgmtNetworkVip: 172.16.5.21
  ServiceVips:
    redis: 172.16.3.98
Comment 7 Marcos Garcia 2017-01-27 01:33:37 UTC 
I have now tried with VIPs outside of the allocation pool, and the deployment worked OK. To be perfectly accurate, ExternalNetworkVIP is actually inside of the allocation pool, whereas all the other ones are now outside of the pool

[stack@undercloud ~]$ cat templates/network-config.yaml 
#This file is an example of an environment file for defining the isolated
#networks and related parameters.
resource_registry:
  # Network Interface templates to use (these files must exist)
  OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute.yaml
  OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml
  OS::TripleO::CephStorage::Net::SoftwareConfig: /home/stack/templates/nic-configs/ceph-storage.yaml
  OS::TripleO::Network::Ports::NetVipMap: /usr/share/openstack-tripleo-heat-templates/network/ports/net_vip_map_external.yaml
  OS::TripleO::Network::Ports::ExternalVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::InternalApiVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::StorageVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::StorageMgmtVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/from_service.yaml

parameter_defaults:
  InternalApiNetCidr: 172.16.3.0/24
  StorageNetCidr: 172.16.4.0/24
  StorageMgmtNetCidr: 172.16.5.0/24
  TenantNetCidr: 172.16.6.0/24
  ExternalNetCidr: 10.12.48.0/24
  # CIDR subnet mask length for provisioning network
  ControlPlaneSubnetCidr: '24'
  # Customize the IP ranges on each network to use for static IPs and VIPs
  InternalApiAllocationPools: [{'start': '172.16.3.20', 'end': '172.16.3.100'}]
  StorageAllocationPools: [{'start': '172.16.4.20', 'end': '172.16.4.100'}]
  StorageMgmtAllocationPools: [{'start': '172.16.5.20', 'end': '172.16.5.100'}]
  TenantAllocationPools: [{'start': '172.16.6.20', 'end': '172.16.6.100'}]
  # Leave room if the external network is also used for floating IPs
  ExternalAllocationPools: [{'start': '10.12.48.100', 'end': '10.12.48.120'}]
  # Gateway router for the external network
  ExternalInterfaceDefaultRoute: 10.12.48.254
  # Gateway router for the provisioning network (or Undercloud IP)
  ControlPlaneDefaultRoute: 192.168.0.1
  # Generally the IP of the Undercloud
  EC2MetadataIp: 192.168.0.1
  # Define the DNS servers (maximum 2) for the overcloud nodes
  DnsServers: ["10.12.48.11","8.8.8.8"]
  # Customize the VLAN IDs to match the local environment
  InternalApiNetworkVlanID: 3 #NOTE: we do not use tags, all vNICs are in native mode
  StorageNetworkVlanID: 4 #NOTE: we do not use tags, all vNICs are in native mode
  StorageMgmtNetworkVlanID: 5 #NOTE: we do not use tags, all vNICs are in native mode
  TenantNetworkVlanID: 6 #NOTE: we do not use tags, all vNICs are in native mode
  # Set to empty string to enable multiple external networks or VLANs
  NeutronExternalNetworkBridge: "br-ex"
  # Customize bonding options, e.g. "mode=4 lacp_rate=1 updelay=1000 miimon=100"
  BondInterfaceOvsOptions: "mode=active-backup"
  ExternalNetworkVip: 10.12.48.100 #must be 100 so DNS can work
  ControlPlaneIP: 192.168.0.4
  InternalApiNetworkVip: 172.16.3.19
  StorageNetworkVip: 172.16.4.19
  StorageMgmtNetworkVip: 172.16.5.19
  ServiceVips:
    redis: 172.16.3.98
Comment 8 curtis 2017-02-03 17:30:38 UTC 
I have also encountered this bug.
Comment 9 Emilien Macchi 2017-10-19 22:33:52 UTC 
Documentation has been provided to not hit this bug:
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html-single/advanced_overcloud_customization/#sect-Predictable_VIPs

Feel free to re-open if there is anything that needs to be fixed in the code.