Description of problem: Trying to set predictable IPs for the management network in OSP8 via environments/ips-from-pool-all.yaml but the IPs assigned in that file are not used. Version-Release number of selected component (if applicable): How reproducible: In environments/ips-from-pool-all.yaml add: OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml and management: - 10.126.168.67 - 10.126.168.68 - 10.126.168.69 to the parameter_defaults: ControllerIPs: section. ensure deployment contains -e /home/stack/tls_templates/environments/ips-from-pool-all.yaml line ensure ManagementNetCidr: 10.126.168.0/25 ManagementAllocationPools: [{'start': '10.126.168.100', 'end': '10.126.168.120'}] is set in referenced template. Steps to Reproduce: 1. modify conifigs as per above 2. deploy with modified ips-from-pool-all.yaml template 3. Actual results: Hosts are built and management nic's created. However assignment comes from ManagementAllocationPools and not ips-from-pool-all.yaml ips listed in management: - 10.126.168.67 - 10.126.168.68 - 10.126.168.69 Expected results: Management nic should be assigned an ip from management: - 10.126.168.67 - 10.126.168.68 - 10.126.168.69 range and not ManagementAllocationPools: [{'start': '10.126.168.100', 'end': '10.126.168.120'}] Additional info:
All other predictable IPs assignments for other networks work correctly. Worth noting that in ips-from-pool-all.yaml we had to add the port assignments for management as they were missing and all others were ther (ie we added OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml, etc)
given the management ips in the ips-from-pool-all.yaml file above we'd expect to see them placed, but instead i see on the controllers: 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 14:02:ec:05:af:14 brd ff:ff:ff:ff:ff:ff inet 172.31.19.84/24 brd 172.31.19.255 scope global eno1 valid_lft forever preferred_lft forever inet 10.126.168.104/25 brd 10.126.168.127 scope global eno1 valid_lft forever preferred_lft forever inet6 fe80::1602:ecff:fe05:af14/64 scope link valid_lft forever preferred_lft forever 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 14:02:ec:05:3c:14 brd ff:ff:ff:ff:ff:ff inet 172.31.19.82/24 brd 172.31.19.255 scope global eno1 valid_lft forever preferred_lft forever inet 10.126.168.101/25 brd 10.126.168.127 scope global eno1 valid_lft forever preferred_lft forever inet6 fe80::1602:ecff:fe05:3c14/64 scope link valid_lft forever preferred_lft forever 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 14:02:ec:05:2c:e8 brd ff:ff:ff:ff:ff:ff inet 172.31.19.80/24 brd 172.31.19.255 scope global eno1 valid_lft forever preferred_lft forever inet 10.126.168.103/25 brd 10.126.168.127 scope global eno1 valid_lft forever preferred_lft forever inet6 fe80::1602:ecff:fe05:2ce8/64 scope link valid_lft forever preferred_lft forever and computes 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 14:02:ec:05:4c:5c brd ff:ff:ff:ff:ff:ff inet 172.31.19.81/24 brd 172.31.19.255 scope global eno1 valid_lft forever preferred_lft forever inet 10.126.168.100/25 brd 10.126.168.127 scope global eno1 valid_lft forever preferred_lft forever inet6 fe80::1602:ecff:fe05:4c5c/64 scope link valid_lft forever preferred_lft forever 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 14:02:ec:05:3c:84 brd ff:ff:ff:ff:ff:ff inet 172.31.19.79/24 brd 172.31.19.255 scope global eno1 valid_lft forever preferred_lft forever inet 10.126.168.102/25 brd 10.126.168.127 scope global eno1 valid_lft forever preferred_lft forever inet6 fe80::1602:ecff:fe05:3c84/64 scope link valid_lft forever preferred_lft forever which is coming from type: interface name: eno1 use_dhcp: false addresses: - ip_netmask: list_join: - '/' - - {get_param: ControlPlaneIp} - {get_param: ControlPlaneSubnetCidr} - ip_netmask: {get_param: ManagementIpSubnet} so the nic is getting built and assigned correctly, but just the wrong ip. and to show the other ips ... here's a nic that does get the right ip from the predictive template: set as internal_api: - 172.31.10.67 - 172.31.10.68 - 172.31.10.69 which yields ... 18: vlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether ce:f5:cc:da:01:e6 brd ff:ff:ff:ff:ff:ff inet 172.31.10.67/24 brd 172.31.10.255 scope global vlan10 valid_lft forever preferred_lft forever inet6 fe80::ccf5:ccff:feda:1e6/64 scope link valid_lft forever preferred_lft forever so the stuff works, but just not management it seems :(
Hi, thanks for pointing out the missing assignments in the env file, Steven just submitted a change to fix that: https://review.openstack.org/#/c/318699 I don't see those changes in the templates attached to the BZ though? Are you providing those mappings from an additional env file at deployment time? Can you attach such an env file in this case? If the same resource is mapped to a different templates there is also a risk that your mapping is overridden so if you could paste the deployment command that would be useful as well. The modifications to the NIC templates look good to me. I will try to reproduce the issue on a test environment and update the BZ with results.
Created attachment 1159539 [details] Replacement environments/ips-from-pool-all.yaml Attached a replacement environments/ips-from-pool-all.yaml that should instantiate the proper ports with predictable IPs.
(In reply to Dan Sneddon from comment #6) > Created attachment 1159539 [details] > Replacement environments/ips-from-pool-all.yaml > > Attached a replacement environments/ips-from-pool-all.yaml that should > instantiate the proper ports with predictable IPs. Actually, after taking a second look at the environments/ips-from-pool-all.yaml I realize that this template was already modified to include the management ports and IPs.
(In reply to Giulio Fidente from comment #5) > Hi, thanks for pointing out the missing assignments in the env file, Steven > just submitted a change to fix that: > > https://review.openstack.org/#/c/318699 > > I don't see those changes in the templates attached to the BZ though? Are > you providing those mappings from an additional env file at deployment time? > Can you attach such an env file in this case? Actually, it looks like these lines were added to ips-from-pool-all.yaml: OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml One thing I noticed in the included tls_templates archive is that I don't see the Management network being initialized. Perhaps there is an environment file being loaded with a -e statement, but I don't see that in any of the custom environment files. Can you please respond with the exact deployment command-line?
It might be worth trying to instantiate the Management network properly by adding this to advanced-networking.yaml in the resource_registry section: OS::TripleO::Network::Management: ../network/management.yaml
Here's the deploy line and relevant bits from the customized files( I think we've got the Management instantiated correctly but we don't do it in the advanced-networking.yaml file): openstack overcloud deploy --templates /home/stack/tls_templates/ \ --ntp-server 10.126.162.9 \ --timeout 60 \ --control-scale 3 \ --compute-scale 2 \ -e /home/stack/tls_templates/advanced-networking.yaml \ -e /home/stack/tls_templates/environments/network-isolation.yaml \ -e /home/stack/tls_templates/environments/ips-from-pool-all.yaml \ -e /home/stack/tls_templates/environments/network-management.yaml \ -e /home/stack/tls_templates/environments/enable-tls.yaml \ -e /home/stack/tls_templates/environments/inject-trust-anchor.yaml \ --neutron-network-vlan-ranges tenant:2500:2599 \ --neutron-flat-networks bond \ --neutron-bridge-mappings bond:br-trunk advanced-networking.yaml only has the one reference to define Mangement: ManagementNetCidr: 10.126.168.0/25 ManagementAllocationPools: [{'start': '10.126.168.100', 'end': '10.126.168.120'}] network-isolation.yaml has these noop'd references: # Management network is optional and disabled by default OS::TripleO::Network::Management: ../network/noop.yaml OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/noop.yaml OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/noop.yaml OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/noop.yaml OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/noop.yaml OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/noop.yaml We manage the predictive pools in ips-from-pool-all.yaml: OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml and we manage all previous noops for the management network in network-management.yaml: OS::TripleO::Network::Management: ../network/management.yaml OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management.yaml OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management.yaml OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management.yaml OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management.yaml OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management.yaml So while we aren't doing it in advanced-networking.yaml we do it, as you mentioned, with addition templates called with -e. And it seems to work as you can see the management network ips are assigned to the nics we want as per the comment 4. From what I can see we have covered everything, yet still not getting the predictive IPs. Should we not use network-management.yaml and just update network-isolation.yaml instead? (ie not have noops?) Does one template override the other in a way I'm not expecting / accounting for?
(In reply to August Simonelli from comment #10) > Here's the deploy line and relevant bits from the customized files( I think > we've got the Management instantiated correctly but we don't do it in the > advanced-networking.yaml file): > > openstack overcloud deploy --templates /home/stack/tls_templates/ \ > --ntp-server 10.126.162.9 \ > --timeout 60 \ > --control-scale 3 \ > --compute-scale 2 \ > -e /home/stack/tls_templates/advanced-networking.yaml \ > -e /home/stack/tls_templates/environments/network-isolation.yaml \ > -e /home/stack/tls_templates/environments/ips-from-pool-all.yaml \ > -e /home/stack/tls_templates/environments/network-management.yaml \ > -e /home/stack/tls_templates/environments/enable-tls.yaml \ > -e /home/stack/tls_templates/environments/inject-trust-anchor.yaml \ > --neutron-network-vlan-ranges tenant:2500:2599 \ > --neutron-flat-networks bond \ > --neutron-bridge-mappings bond:br-trunk > > advanced-networking.yaml only has the one reference to define Mangement: The issue is that when you use -e includes, the later includes take precedent. So in advanced-networking.yaml the ManagementPort is set to management-from-pool.yaml, but then network-management.yaml is included after that and sets those ports to management.yaml, which uses the AllocationPools. You have two choices: 1) Include advanced-networking.yaml after network-management.yaml. (actually, we generally recommend that built-in templates be included before custom templates) 2) Don't include network-management.yaml at all, and instead include the one pertinent line that instantiates the Management network into advanced-networking.yaml in the resource_registry section: OS::TripleO::Network::Management: /path/to/network/management.yaml
Gotcha. That's what I was wondering about overriding templates. Ok, for my own clarity this is what i understand: We get the management network ok cause we stay consistent with: OS::TripleO::Network::Management: ../network/management.yaml but we override our ManagementPort settings we have in ips-from-pool-all.yaml OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml when we call network-management.yaml with its duplicate ManagementPort pointing to the non pool files OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management.yaml This makes sense! Thank you. We will try again with your suggestions.
We can't get this to work even when following your advice. Here's what I've done: Given this deploy line: openstack overcloud deploy --templates /home/stack/tls_templates_august/ \ --ntp-server 10.126.162.9 \ --timeout 60 \ --control-scale 3 \ --compute-scale 2 \ -e /home/stack/tls_templates_august/environments/network-isolation.yaml \ -e /home/stack/tls_templates_august/environments/network-management.yaml \ -e /home/stack/tls_templates_august/environments/ips-from-pool-all.yaml \ -e /home/stack/tls_templates_august/environments/enable-tls.yaml \ -e /home/stack/tls_templates_august/environments/inject-trust-anchor.yaml \ -e /home/stack/tls_templates_august/advanced-networking.yaml \ --neutron-network-vlan-ranges tenant:2500:2599 \ --neutron-flat-networks bond \ --neutron-bridge-mappings bond:br-trunk network-isolation.yaml disables all Management networks and ports by default: $ grep Management environments/network-isolation.yaml # Management network is optional and disabled by default OS::TripleO::Network::Management: ../network/noop.yaml OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/noop.yaml OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/noop.yaml OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/noop.yaml OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/noop.yaml OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/noop.yaml network-management.yaml enables all Management networks and ports but WITHOUT predictive IPs: $ grep Management environments/network-management.yaml OS::TripleO::Network::Management: ../network/management.yaml OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management.yaml OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management.yaml OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management.yaml OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management.yaml OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management.yaml ips-from-pool-all.yaml turns on predictive IPs for all networks, including management: $ grep Management environments/ips-from-pool-all.yaml OS::TripleO::Controller::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::CephStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::SwiftStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml OS::TripleO::BlockStorage::Ports::ManagementPort: ../network/ports/management_from_pool.yaml note it does not set OS::TripleO::Network::Management: in anyway, network-management.yaml does that. advanced-networking.yaml does not set not set or enable ANY management networks or ports, it only sets ManagementNetCidr and ManagementAllocationPools $ grep Management advanced-networking.yaml ManagementNetCidr: 10.126.168.0/25 ManagementAllocationPools: [{'start': '10.126.168.100', 'end': '10.126.168.120'}] but this ends with: 2016-05-22 03:21:06 [UpdateDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded 2016-05-22 03:21:06 [UpdateDeployment]: CREATE_COMPLETE state changed 2016-05-22 03:21:06 [overcloud-Compute-44uci3xoib3r-0-jo7du436bhf2]: CREATE_FAILED Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect. 2016-05-22 03:21:07 [0]: CREATE_FAILED resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect. 2016-05-22 03:21:38 [UpdateDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded 2016-05-22 03:21:39 [UpdateDeployment]: CREATE_COMPLETE state changed 2016-05-22 03:21:39 [overcloud-Controller-fjqy4py5knfy-0-ssr2azkj35sc]: CREATE_FAILED Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect. 2016-05-22 03:21:40 [0]: CREATE_FAILED resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect. 2016-05-22 03:25:07 [1]: CREATE_FAILED CREATE aborted 2016-05-22 03:25:07 [overcloud-Compute-44uci3xoib3r]: UPDATE_FAILED resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect. 2016-05-22 03:25:08 [Compute]: CREATE_FAILED resources.Compute: resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect. 2016-05-22 03:25:41 [1]: CREATE_FAILED CREATE aborted 2016-05-22 03:25:41 [2]: CREATE_FAILED CREATE aborted 2016-05-22 03:25:41 [overcloud-Controller-fjqy4py5knfy]: UPDATE_FAILED resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect. 2016-05-22 03:25:42 [Controller]: CREATE_FAILED resources.Controller: resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect. 2016-05-22 03:25:43 [overcloud]: CREATE_FAILED Resource CREATE failed: resources.Compute: resources[0]: Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect. Stack overcloud CREATE_FAILED
following this https://bugs.launchpad.net/tripleo/+bug/1575622 gets me past that but still have issues. might not worry about mgmt network.
The issue with the str_split function should be fixed by https://review.openstack.org/319840 This change, together with the environments file ordering issue should make the static assignments for 'management' to work.
Yes, looks good. Thanks!
Related to this BZ, I just submitted an upstream review that will add the ability to pre-select IPv6 IPs on the Management network if it is IPv6 and not IPv4. The ips-from-pool-all.yaml only implements IPv4, but the same technique should work if you create a custom version of that template and point to management_from_pool_v6.yaml (once this change has merged): https://review.openstack.org/#/c/322817/
Closing this as fixes merged to stable branches over 16 months ago.