Bug 1337209 - setting predictable IPs for managment network in ips-from-pool-all.yaml doesn't work
Summary: setting predictable IPs for managment network in ips-from-pool-all.yaml doesn...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 8.0 (Liberty)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: async
: 8.0 (Liberty)
Assignee: Giulio Fidente
QA Contact: Arik Chernetsky
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-05-18 13:48 UTC by August Simonelli
Modified: 2019-10-10 12:07 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-13 20:39:52 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Replacement environments/ips-from-pool-all.yaml (3.86 KB, text/plain)
2016-05-19 16:26 UTC, Dan Sneddon
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 318699 0 None MERGED Add Management net to ips-from-pool-all example 2020-04-04 15:16:20 UTC
OpenStack gerrit 319840 0 None MERGED Update management_from_pool template version to 2015-10-15 2020-04-04 15:16:20 UTC
Red Hat Knowledge Base (Solution) 2338631 0 None None None 2016-05-26 06:23:34 UTC

Description August Simonelli 2016-05-18 13:48:08 UTC
Description of problem:
Trying to set predictable IPs for the management network in OSP8 via environments/ips-from-pool-all.yaml but the IPs assigned in that file are not used.

Version-Release number of selected component (if applicable):


How reproducible:

In environments/ips-from-pool-all.yaml add:
  OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml

and

   management:
    - 10.126.168.67
    - 10.126.168.68
    - 10.126.168.69

to the 

parameter_defaults:
  ControllerIPs:

section.

ensure deployment contains

 -e /home/stack/tls_templates/environments/ips-from-pool-all.yaml

line

ensure

  ManagementNetCidr: 10.126.168.0/25
  ManagementAllocationPools: [{'start': '10.126.168.100', 'end': '10.126.168.120'}]

is set in referenced template.

Steps to Reproduce:
1. modify conifigs as per above
2. deploy with modified ips-from-pool-all.yaml template
3.

Actual results:

Hosts are built and management nic's created. However assignment comes from ManagementAllocationPools and not ips-from-pool-all.yaml ips listed in 

   management:
    - 10.126.168.67
    - 10.126.168.68
    - 10.126.168.69


Expected results:

Management nic should be assigned an ip from 

   management:
    - 10.126.168.67
    - 10.126.168.68
    - 10.126.168.69

range and not

 ManagementAllocationPools: [{'start': '10.126.168.100', 'end': '10.126.168.120'}]


Additional info:

Comment 2 August Simonelli 2016-05-18 13:52:22 UTC
All other predictable IPs assignments for other networks work correctly. Worth noting that in ips-from-pool-all.yaml we had to add the port assignments for management as they were missing and all others were ther (ie we added   OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml, etc)

Comment 4 August Simonelli 2016-05-19 10:02:05 UTC
given the management ips in the ips-from-pool-all.yaml file above we'd expect to see them placed, but instead i see on the controllers:

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 14:02:ec:05:af:14 brd ff:ff:ff:ff:ff:ff
    inet 172.31.19.84/24 brd 172.31.19.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet 10.126.168.104/25 brd 10.126.168.127 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::1602:ecff:fe05:af14/64 scope link
       valid_lft forever preferred_lft forever


2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 14:02:ec:05:3c:14 brd ff:ff:ff:ff:ff:ff
    inet 172.31.19.82/24 brd 172.31.19.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet 10.126.168.101/25 brd 10.126.168.127 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::1602:ecff:fe05:3c14/64 scope link
       valid_lft forever preferred_lft forever

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 14:02:ec:05:2c:e8 brd ff:ff:ff:ff:ff:ff
    inet 172.31.19.80/24 brd 172.31.19.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet 10.126.168.103/25 brd 10.126.168.127 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::1602:ecff:fe05:2ce8/64 scope link
       valid_lft forever preferred_lft forever

and computes

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 14:02:ec:05:4c:5c brd ff:ff:ff:ff:ff:ff
    inet 172.31.19.81/24 brd 172.31.19.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet 10.126.168.100/25 brd 10.126.168.127 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::1602:ecff:fe05:4c5c/64 scope link
       valid_lft forever preferred_lft forever


2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 14:02:ec:05:3c:84 brd ff:ff:ff:ff:ff:ff
    inet 172.31.19.79/24 brd 172.31.19.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet 10.126.168.102/25 brd 10.126.168.127 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::1602:ecff:fe05:3c84/64 scope link
       valid_lft forever preferred_lft forever

which is coming from

              type: interface
              name: eno1
              use_dhcp: false
              addresses:
                -
                  ip_netmask:
                    list_join:
                      - '/'
                      - - {get_param: ControlPlaneIp}
                        - {get_param: ControlPlaneSubnetCidr}
                -
                  ip_netmask: {get_param: ManagementIpSubnet}

so the nic is getting built and assigned correctly, but just the wrong ip.

and to show the other ips ... here's a nic that does get the right ip from the predictive template:

set as 

    internal_api:
    - 172.31.10.67
    - 172.31.10.68
    - 172.31.10.69

which yields ...

18: vlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether ce:f5:cc:da:01:e6 brd ff:ff:ff:ff:ff:ff
    inet 172.31.10.67/24 brd 172.31.10.255 scope global vlan10
       valid_lft forever preferred_lft forever
    inet6 fe80::ccf5:ccff:feda:1e6/64 scope link
       valid_lft forever preferred_lft forever

so the stuff works, but just not management it seems :(

Comment 5 Giulio Fidente 2016-05-19 14:50:53 UTC
Hi, thanks for pointing out the missing assignments in the env file, Steven just submitted a change to fix that:

https://review.openstack.org/#/c/318699

I don't see those changes in the templates attached to the BZ though? Are you providing those mappings from an additional env file at deployment time? Can you attach such an env file in this case?

If the same resource is mapped to a different templates there is also a risk that your mapping is overridden so if you could paste the deployment command that would be useful as well.

The modifications to the NIC templates look good to me.

I will try to reproduce the issue on a test environment and update the BZ with results.

Comment 6 Dan Sneddon 2016-05-19 16:26:36 UTC
Created attachment 1159539 [details]
Replacement environments/ips-from-pool-all.yaml

Attached a replacement environments/ips-from-pool-all.yaml that should instantiate the proper ports with predictable IPs.

Comment 7 Dan Sneddon 2016-05-19 16:34:31 UTC
(In reply to Dan Sneddon from comment #6)
> Created attachment 1159539 [details]
> Replacement environments/ips-from-pool-all.yaml
> 
> Attached a replacement environments/ips-from-pool-all.yaml that should
> instantiate the proper ports with predictable IPs.

Actually, after taking a second look at the environments/ips-from-pool-all.yaml I realize that this template was already modified to include the management ports and IPs.

Comment 8 Dan Sneddon 2016-05-19 16:48:14 UTC
(In reply to Giulio Fidente from comment #5)
> Hi, thanks for pointing out the missing assignments in the env file, Steven
> just submitted a change to fix that:
> 
> https://review.openstack.org/#/c/318699
> 
> I don't see those changes in the templates attached to the BZ though? Are
> you providing those mappings from an additional env file at deployment time?
> Can you attach such an env file in this case?

Actually, it looks like these lines were added to ips-from-pool-all.yaml:

  OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml

One thing I noticed in the included tls_templates archive is that I don't see the Management network being initialized. Perhaps there is an environment file being loaded with a -e statement, but I don't see that in any of the custom environment files.

Can you please respond with the exact deployment command-line?

Comment 9 Dan Sneddon 2016-05-19 16:50:44 UTC
It might be worth trying to instantiate the Management network properly by adding this to advanced-networking.yaml in the resource_registry section:

  OS::TripleO::Network::Management: ../network/management.yaml

Comment 10 August Simonelli 2016-05-19 21:43:37 UTC
Here's the deploy line and relevant bits from the customized files( I think we've got the Management instantiated correctly but we don't do it in the advanced-networking.yaml file):

openstack overcloud deploy --templates /home/stack/tls_templates/ \
 --ntp-server 10.126.162.9 \
 --timeout 60 \
 --control-scale 3 \
 --compute-scale 2 \
 -e /home/stack/tls_templates/advanced-networking.yaml \
 -e /home/stack/tls_templates/environments/network-isolation.yaml \
 -e /home/stack/tls_templates/environments/ips-from-pool-all.yaml \
 -e /home/stack/tls_templates/environments/network-management.yaml \
 -e /home/stack/tls_templates/environments/enable-tls.yaml \
 -e /home/stack/tls_templates/environments/inject-trust-anchor.yaml \
  --neutron-network-vlan-ranges tenant:2500:2599 \
 --neutron-flat-networks bond \
 --neutron-bridge-mappings bond:br-trunk

advanced-networking.yaml only has the one reference to define Mangement:

  ManagementNetCidr: 10.126.168.0/25
  ManagementAllocationPools: [{'start': '10.126.168.100', 'end': '10.126.168.120'}]

network-isolation.yaml has these noop'd references:

  # Management network is optional and disabled by default
  OS::TripleO::Network::Management: ../network/noop.yaml
  OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/noop.yaml

We manage the predictive pools in ips-from-pool-all.yaml: 

  OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml

and we manage all previous noops for the management network in network-management.yaml:

  OS::TripleO::Network::Management: ../network/management.yaml
  OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management.yaml

So while we aren't doing it in advanced-networking.yaml we do it, as you mentioned, with addition templates called with -e. And it seems to work as you can see the management network ips are assigned to the nics we want as per the comment 4.

From what I can see we have covered everything, yet still not getting the predictive IPs. Should we not use network-management.yaml and just update network-isolation.yaml instead? (ie not have noops?) Does one template override the other in a way I'm not expecting / accounting for?

Comment 13 Dan Sneddon 2016-05-19 21:53:44 UTC
(In reply to August Simonelli from comment #10)
> Here's the deploy line and relevant bits from the customized files( I think
> we've got the Management instantiated correctly but we don't do it in the
> advanced-networking.yaml file):
> 
> openstack overcloud deploy --templates /home/stack/tls_templates/ \
>  --ntp-server 10.126.162.9 \
>  --timeout 60 \
>  --control-scale 3 \
>  --compute-scale 2 \
>  -e /home/stack/tls_templates/advanced-networking.yaml \
>  -e /home/stack/tls_templates/environments/network-isolation.yaml \
>  -e /home/stack/tls_templates/environments/ips-from-pool-all.yaml \
>  -e /home/stack/tls_templates/environments/network-management.yaml \
>  -e /home/stack/tls_templates/environments/enable-tls.yaml \
>  -e /home/stack/tls_templates/environments/inject-trust-anchor.yaml \
>   --neutron-network-vlan-ranges tenant:2500:2599 \
>  --neutron-flat-networks bond \
>  --neutron-bridge-mappings bond:br-trunk
> 
> advanced-networking.yaml only has the one reference to define Mangement:

The issue is that when you use -e includes, the later includes take precedent. So in advanced-networking.yaml the ManagementPort is set to management-from-pool.yaml, but then network-management.yaml is included after that and sets those ports to management.yaml, which uses the AllocationPools.

You have two choices:

1) Include advanced-networking.yaml after network-management.yaml. (actually, we generally recommend that built-in templates be included before custom templates)

2) Don't include network-management.yaml at all, and instead include the one pertinent line that instantiates the Management network into advanced-networking.yaml in the resource_registry section:
  OS::TripleO::Network::Management: /path/to/network/management.yaml

Comment 14 August Simonelli 2016-05-19 22:07:58 UTC
Gotcha. That's what I was wondering about overriding templates. Ok, for my own clarity this is what i understand:

We get the management network ok cause we stay consistent with:

OS::TripleO::Network::Management: ../network/management.yaml

but we override our ManagementPort settings we have in ips-from-pool-all.yaml

 OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml

when we call network-management.yaml with its duplicate ManagementPort pointing to the non pool files

OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management.yaml

This makes sense! Thank you. We will try again with your suggestions.

Comment 15 August Simonelli 2016-05-22 03:29:18 UTC
We can't get this to work even when following your advice. Here's what I've done:

Given this deploy line:

openstack overcloud deploy --templates /home/stack/tls_templates_august/ \
 --ntp-server 10.126.162.9 \
 --timeout 60 \
 --control-scale 3 \
 --compute-scale 2 \
 -e /home/stack/tls_templates_august/environments/network-isolation.yaml \
 -e /home/stack/tls_templates_august/environments/network-management.yaml \
 -e /home/stack/tls_templates_august/environments/ips-from-pool-all.yaml \
 -e /home/stack/tls_templates_august/environments/enable-tls.yaml \
 -e /home/stack/tls_templates_august/environments/inject-trust-anchor.yaml \
 -e /home/stack/tls_templates_august/advanced-networking.yaml \
  --neutron-network-vlan-ranges tenant:2500:2599 \
 --neutron-flat-networks bond \
 --neutron-bridge-mappings bond:br-trunk

network-isolation.yaml disables all Management networks and ports by default:

$ grep Management environments/network-isolation.yaml
  # Management network is optional and disabled by default
  OS::TripleO::Network::Management: ../network/noop.yaml
  OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/noop.yaml
  OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/noop.yaml

network-management.yaml enables all Management networks and ports but WITHOUT
predictive IPs:

$ grep Management environments/network-management.yaml
  OS::TripleO::Network::Management: ../network/management.yaml
  OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management.yaml
  OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management.yaml

ips-from-pool-all.yaml turns on predictive IPs for all networks, including
management:

$ grep Management environments/ips-from-pool-all.yaml
  OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml
  OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml

note it does not set OS::TripleO::Network::Management: in anyway, network-management.yaml does that.

advanced-networking.yaml does not set not set or enable ANY management networks or ports, it only sets ManagementNetCidr and
ManagementAllocationPools

$ grep Management advanced-networking.yaml
  ManagementNetCidr: 10.126.168.0/25
  ManagementAllocationPools: [{'start': '10.126.168.100', 'end': '10.126.168.120'}]

but this ends with:

2016-05-22 03:21:06 [UpdateDeployment]: SIGNAL_IN_PROGRESS  Signal: deployment succeeded
2016-05-22 03:21:06 [UpdateDeployment]: CREATE_COMPLETE  state changed
2016-05-22 03:21:06 [overcloud-Compute-44uci3xoib3r-0-jo7du436bhf2]: CREATE_FAILED  Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:21:07 [0]: CREATE_FAILED  resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:21:38 [UpdateDeployment]: SIGNAL_IN_PROGRESS  Signal: deployment succeeded
2016-05-22 03:21:39 [UpdateDeployment]: CREATE_COMPLETE  state changed
2016-05-22 03:21:39 [overcloud-Controller-fjqy4py5knfy-0-ssr2azkj35sc]: CREATE_FAILED  Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:21:40 [0]: CREATE_FAILED  resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:25:07 [1]: CREATE_FAILED  CREATE aborted
2016-05-22 03:25:07 [overcloud-Compute-44uci3xoib3r]: UPDATE_FAILED  resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:25:08 [Compute]: CREATE_FAILED  resources.Compute: resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:25:41 [1]: CREATE_FAILED  CREATE aborted
2016-05-22 03:25:41 [2]: CREATE_FAILED  CREATE aborted
2016-05-22 03:25:41 [overcloud-Controller-fjqy4py5knfy]: UPDATE_FAILED  resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:25:42 [Controller]: CREATE_FAILED  resources.Controller: resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
2016-05-22 03:25:43 [overcloud]: CREATE_FAILED  Resource CREATE failed: resources.Compute: resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.
Stack overcloud CREATE_FAILED

Comment 16 August Simonelli 2016-05-22 04:52:52 UTC
following this https://bugs.launchpad.net/tripleo/+bug/1575622 gets me past that but still have issues. might not worry about mgmt network.

Comment 17 Giulio Fidente 2016-05-23 10:32:38 UTC
The issue with the str_split function should be fixed by https://review.openstack.org/319840

This change, together with the environments file ordering issue should make the static assignments for 'management' to work.

Comment 18 August Simonelli 2016-05-23 10:42:00 UTC
Yes, looks good. Thanks!

Comment 19 Dan Sneddon 2016-06-08 17:56:01 UTC
Related to this BZ, I just submitted an upstream review that will add the ability to pre-select IPv6 IPs on the Management network if it is IPv6 and not IPv4. The ips-from-pool-all.yaml only implements IPv4, but the same technique should work if you create a custom version of that template and point to management_from_pool_v6.yaml (once this change has merged):

https://review.openstack.org/#/c/322817/

Comment 21 Bob Fournier 2018-02-13 20:39:52 UTC
Closing this as fixes merged to stable branches over 16 months ago.


Note You need to log in before you can comment on or make changes to this bug.