All other predictable IPs assignments for other networks work correctly. Worth noting that in ips-from-pool-all.yaml we had to add the port assignments for management as they were missing and all others were ther (ie we added   OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml, etc)

given the management ips in the ips-from-pool-all.yaml file above we'd expect to see them placed, but instead i see on the controllers:

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 14:02:ec:05:af:14 brd ff:ff:ff:ff:ff:ff
    inet 172.31.19.84/24 brd 172.31.19.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet 10.126.168.104/25 brd 10.126.168.127 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::1602:ecff:fe05:af14/64 scope link
       valid_lft forever preferred_lft forever


2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 14:02:ec:05:3c:14 brd ff:ff:ff:ff:ff:ff
    inet 172.31.19.82/24 brd 172.31.19.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet 10.126.168.101/25 brd 10.126.168.127 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::1602:ecff:fe05:3c14/64 scope link
       valid_lft forever preferred_lft forever

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 14:02:ec:05:2c:e8 brd ff:ff:ff:ff:ff:ff
    inet 172.31.19.80/24 brd 172.31.19.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet 10.126.168.103/25 brd 10.126.168.127 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::1602:ecff:fe05:2ce8/64 scope link
       valid_lft forever preferred_lft forever

and computes

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 14:02:ec:05:4c:5c brd ff:ff:ff:ff:ff:ff
    inet 172.31.19.81/24 brd 172.31.19.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet 10.126.168.100/25 brd 10.126.168.127 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::1602:ecff:fe05:4c5c/64 scope link
       valid_lft forever preferred_lft forever


2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 14:02:ec:05:3c:84 brd ff:ff:ff:ff:ff:ff
    inet 172.31.19.79/24 brd 172.31.19.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet 10.126.168.102/25 brd 10.126.168.127 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::1602:ecff:fe05:3c84/64 scope link
       valid_lft forever preferred_lft forever

which is coming from

              type: interface
              name: eno1
              use_dhcp: false
              addresses:
                -
                  ip_netmask:
                    list_join:
                      - '/'
                      - - {get_param: ControlPlaneIp}
                        - {get_param: ControlPlaneSubnetCidr}
                -
                  ip_netmask: {get_param: ManagementIpSubnet}

so the nic is getting built and assigned correctly, but just the wrong ip.

and to show the other ips ... here's a nic that does get the right ip from the predictive template:

set as 

    internal_api:
    - 172.31.10.67
    - 172.31.10.68
    - 172.31.10.69

which yields ...

18: vlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether ce:f5:cc:da:01:e6 brd ff:ff:ff:ff:ff:ff
    inet 172.31.10.67/24 brd 172.31.10.255 scope global vlan10
       valid_lft forever preferred_lft forever
    inet6 fe80::ccf5:ccff:feda:1e6/64 scope link
       valid_lft forever preferred_lft forever

so the stuff works, but just not management it seems :(

Hi, thanks for pointing out the missing assignments in the env file, Steven just submitted a change to fix that:

https://review.openstack.org/#/c/318699

I don't see those changes in the templates attached to the BZ though? Are you providing those mappings from an additional env file at deployment time? Can you attach such an env file in this case?

If the same resource is mapped to a different templates there is also a risk that your mapping is overridden so if you could paste the deployment command that would be useful as well.

The modifications to the NIC templates look good to me.

I will try to reproduce the issue on a test environment and update the BZ with results.

Created attachment 1159539 [details]
Replacement environments/ips-from-pool-all.yaml

Attached a replacement environments/ips-from-pool-all.yaml that should instantiate the proper ports with predictable IPs.

(In reply to Dan Sneddon from comment #6)
> Created attachment 1159539 [details]
> Replacement environments/ips-from-pool-all.yaml
> 
> Attached a replacement environments/ips-from-pool-all.yaml that should
> instantiate the proper ports with predictable IPs.

Actually, after taking a second look at the environments/ips-from-pool-all.yaml I realize that this template was already modified to include the management ports and IPs.

(In reply to Giulio Fidente from comment #5)
> Hi, thanks for pointing out the missing assignments in the env file, Steven
> just submitted a change to fix that:
> 
> https://review.openstack.org/#/c/318699
> 
> I don't see those changes in the templates attached to the BZ though? Are
> you providing those mappings from an additional env file at deployment time?
> Can you attach such an env file in this case?

Actually, it looks like these lines were added to ips-from-pool-all.yaml:

  OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml

One thing I noticed in the included tls_templates archive is that I don't see the Management network being initialized. Perhaps there is an environment file being loaded with a -e statement, but I don't see that in any of the custom environment files.

Can you please respond with the exact deployment command-line?

It might be worth trying to instantiate the Management network properly by adding this to advanced-networking.yaml in the resource_registry section:

  OS::TripleO::Network::Management: ../network/management.yaml

Here's the deploy line and relevant bits from the customized files( I think we've got the Management instantiated correctly but we don't do it in the advanced-networking.yaml file):

openstack overcloud deploy --templates /home/stack/tls_templates/ \
 --ntp-server 10.126.162.9 \
 --timeout 60 \
 --control-scale 3 \
 --compute-scale 2 \
 -e /home/stack/tls_templates/advanced-networking.yaml \
 -e /home/stack/tls_templates/environments/network-isolation.yaml \
 -e /home/stack/tls_templates/environments/ips-from-pool-all.yaml \
 -e /home/stack/tls_templates/environments/network-management.yaml \
 -e /home/stack/tls_templates/environments/enable-tls.yaml \
 -e /home/stack/tls_templates/environments/inject-trust-anchor.yaml \
  --neutron-network-vlan-ranges tenant:2500:2599 \
 --neutron-flat-networks bond \
 --neutron-bridge-mappings bond:br-trunk

advanced-networking.yaml only has the one reference to define Mangement:

  ManagementNetCidr: 10.126.168.0/25
  ManagementAllocationPools: [{'start': '10.126.168.100', 'end': '10.126.168.120'}]

network-isolation.yaml has these noop'd references:

  # Management network is optional and disabled by default
  OS::TripleO::Network::Management: ../network/noop.yaml
  OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/noop.yaml

We manage the predictive pools in ips-from-pool-all.yaml: 

  OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml

and we manage all previous noops for the management network in network-management.yaml:

  OS::TripleO::Network::Management: ../network/management.yaml
  OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management.yaml

So while we aren't doing it in advanced-networking.yaml we do it, as you mentioned, with addition templates called with -e. And it seems to work as you can see the management network ips are assigned to the nics we want as per the comment 4.

From what I can see we have covered everything, yet still not getting the predictive IPs. Should we not use network-management.yaml and just update network-isolation.yaml instead? (ie not have noops?) Does one template override the other in a way I'm not expecting / accounting for?

(In reply to August Simonelli from comment #10)
> Here's the deploy line and relevant bits from the customized files( I think
> we've got the Management instantiated correctly but we don't do it in the
> advanced-networking.yaml file):
> 
> openstack overcloud deploy --templates /home/stack/tls_templates/ \
>  --ntp-server 10.126.162.9 \
>  --timeout 60 \
>  --control-scale 3 \
>  --compute-scale 2 \
>  -e /home/stack/tls_templates/advanced-networking.yaml \
>  -e /home/stack/tls_templates/environments/network-isolation.yaml \
>  -e /home/stack/tls_templates/environments/ips-from-pool-all.yaml \
>  -e /home/stack/tls_templates/environments/network-management.yaml \
>  -e /home/stack/tls_templates/environments/enable-tls.yaml \
>  -e /home/stack/tls_templates/environments/inject-trust-anchor.yaml \
>   --neutron-network-vlan-ranges tenant:2500:2599 \
>  --neutron-flat-networks bond \
>  --neutron-bridge-mappings bond:br-trunk
> 
> advanced-networking.yaml only has the one reference to define Mangement:

The issue is that when you use -e includes, the later includes take precedent. So in advanced-networking.yaml the ManagementPort is set to management-from-pool.yaml, but then network-management.yaml is included after that and sets those ports to management.yaml, which uses the AllocationPools.

You have two choices:

1) Include advanced-networking.yaml after network-management.yaml. (actually, we generally recommend that built-in templates be included before custom templates)

2) Don't include network-management.yaml at all, and instead include the one pertinent line that instantiates the Management network into advanced-networking.yaml in the resource_registry section:
  OS::TripleO::Network::Management: /path/to/network/management.yaml

Gotcha. That's what I was wondering about overriding templates. Ok, for my own clarity this is what i understand:

We get the management network ok cause we stay consistent with:

OS::TripleO::Network::Management: ../network/management.yaml

but we override our ManagementPort settings we have in ips-from-pool-all.yaml

 OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml

when we call network-management.yaml with its duplicate ManagementPort pointing to the non pool files

OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management.yaml

This makes sense! Thank you. We will try again with your suggestions.

We can't get this to work even when following your advice. Here's what I've done:

Given this deploy line:

openstack overcloud deploy --templates /home/stack/tls_templates_august/ \
 --ntp-server 10.126.162.9 \
 --timeout 60 \
 --control-scale 3 \
 --compute-scale 2 \
 -e /home/stack/tls_templates_august/environments/network-isolation.yaml \
 -e /home/stack/tls_templates_august/environments/network-management.yaml \
 -e /home/stack/tls_templates_august/environments/ips-from-pool-all.yaml \
 -e /home/stack/tls_templates_august/environments/enable-tls.yaml \
 -e /home/stack/tls_templates_august/environments/inject-trust-anchor.yaml \
 -e /home/stack/tls_templates_august/advanced-networking.yaml \
  --neutron-network-vlan-ranges tenant:2500:2599 \
 --neutron-flat-networks bond \
 --neutron-bridge-mappings bond:br-trunk

network-isolation.yaml disables all Management networks and ports by default:

$ grep Management environments/network-isolation.yaml
  # Management network is optional and disabled by default
  OS::TripleO::Network::Management: ../network/noop.yaml
  OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/noop.yaml

network-management.yaml enables all Management networks and ports but WITHOUT
predictive IPs:

$ grep Management environments/network-management.yaml
  OS::TripleO::Network::Management: ../network/management.yaml
  OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management.yaml

ips-from-pool-all.yaml turns on predictive IPs for all networks, including
management:

$ grep Management environments/ips-from-pool-all.yaml
  OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml

note it does not set OS::TripleO::Network::Management: in anyway, network-management.yaml does that.

advanced-networking.yaml does not set not set or enable ANY management networks or ports, it only sets ManagementNetCidr and
ManagementAllocationPools

$ grep Management advanced-networking.yaml
  ManagementNetCidr: 10.126.168.0/25
  ManagementAllocationPools: [{'start': '10.126.168.100', 'end': '10.126.168.120'}]

but this ends with:

2016-05-22 03:21:06 [UpdateDeployment]: SIGNAL_IN_PROGRESS  Signal: deployment succeeded
2016-05-22 03:21:06 [UpdateDeployment]: CREATE_COMPLETE  state changed
2016-05-22 03:21:06 [overcloud-Compute-44uci3xoib3r-0-jo7du436bhf2]: CREATE_FAILED  Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:21:07 [0]: CREATE_FAILED  resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:21:38 [UpdateDeployment]: SIGNAL_IN_PROGRESS  Signal: deployment succeeded
2016-05-22 03:21:39 [UpdateDeployment]: CREATE_COMPLETE  state changed
2016-05-22 03:21:39 [overcloud-Controller-fjqy4py5knfy-0-ssr2azkj35sc]: CREATE_FAILED  Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:21:40 [0]: CREATE_FAILED  resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:25:07 [1]: CREATE_FAILED  CREATE aborted
2016-05-22 03:25:07 [overcloud-Compute-44uci3xoib3r]: UPDATE_FAILED  resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:25:08 [Compute]: CREATE_FAILED  resources.Compute: resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:25:41 [1]: CREATE_FAILED  CREATE aborted
2016-05-22 03:25:41 [2]: CREATE_FAILED  CREATE aborted
2016-05-22 03:25:41 [overcloud-Controller-fjqy4py5knfy]: UPDATE_FAILED  resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:25:42 [Controller]: CREATE_FAILED  resources.Controller: resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:25:43 [overcloud]: CREATE_FAILED  Resource CREATE failed: resources.Compute: resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
Stack overcloud CREATE_FAILED

following this https://bugs.launchpad.net/tripleo/+bug/1575622 gets me past that but still have issues. might not worry about mgmt network.

The issue with the str_split function should be fixed by https://review.openstack.org/319840

This change, together with the environments file ordering issue should make the static assignments for 'management' to work.

Yes, looks good. Thanks!

Related to this BZ, I just submitted an upstream review that will add the ability to pre-select IPv6 IPs on the Management network if it is IPv6 and not IPv4. The ips-from-pool-all.yaml only implements IPv4, but the same technique should work if you create a custom version of that template and point to management_from_pool_v6.yaml (once this change has merged):

https://review.openstack.org/#/c/322817/

Closing this as fixes merged to stable branches over 16 months ago.