Bug 1672407

Summary: [Hackfest] Predictable IP's working for ControllerIPs, but not ComputeIPs
Product: Red Hat OpenStack Reporter: Ken Savich <ksavich>
Component: openstack-tripleo-heat-templatesAssignee: Emilien Macchi <emacchi>
Status: CLOSED DUPLICATE QA Contact: Gurenko Alex <agurenko>
Severity: medium Docs Contact:
Priority: medium    
Version: 14.0 (Rocky)CC: aschultz, bfournie, cpaquin, dsneddon, hjensas, kholden, ksavich, lmarsh, mburns
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-02-14 21:07:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ken Savich 2019-02-04 21:05:20 UTC
Description of problem:

Deploying OSP 14 with predictable IPs on the control plane works for Controllers, but not Compute nodes in the overcloud.

Version-Release number of selected component (if applicable):

14

How reproducible:

100%

Steps to Reproduce:
1.Create ips-from-pool-ctlplane.yaml
2.Include ips-from-pool-ctlplane.yaml in overcloud deploy command
3.Check IP of of overcloud nodes

Actual results:

Controllers get the supplied IP addresses, while Compute node(s) do not.


Expected results:

All IPs in ips-from-pool-ctlplane.yaml should be assigned to their roles specified

Additional info:

ips-from-pool-ctlplane.yaml:

parameter_defaults:
  ControllerIPs:
    - 10.1.1.10
    - 10.1.1.11
    - 10.1.1.13
  ComputeIPs:
    ctlplane:
    - 10.1.1.20
---
Control Plane is using 10.1.1.0/24

---

# output of `openstack server list` - note, compute node has 10.1.1.23, not 10.1.1.20

(undercloud) [stack@director templates]$ openstack server list
+--------------------------------------+------------------------+--------+--------------------+----------------+---------+
| ID                                   | Name                   | Status | Networks           | Image          | Flavor  |
+--------------------------------------+------------------------+--------+--------------------+----------------+---------+
| 60cf9ea8-6b5a-475f-bb0e-ca389a53737d | overcloud-controller-2 | ACTIVE | ctlplane=10.1.1.13 | overcloud-full | control |
| 19011d7f-5210-43cd-999a-01f2163b6981 | overcloud-controller-1 | ACTIVE | ctlplane=10.1.1.11 | overcloud-full | control |
| 60810e29-122c-4bdb-9050-82b04f4b5b1e | overcloud-controller-0 | ACTIVE | ctlplane=10.1.1.10 | overcloud-full | control |
| 79b0ee46-1483-49c3-9733-34087c132ea3 | overcloud-compute-0    | ACTIVE | ctlplane=10.1.1.22 | overcloud-full | compute |

Comment 1 Ken Holden 2019-02-05 21:48:03 UTC
experienced this too.  deployed with 3 roles (Controller, ComputeHci, and Compute).  all controllers and HCI nodes deployed with static ctlplane IP, but the Compute node grabbed it's ctlplane IP from the DHCP pool in director.  

# Environment file demonstrating how to pre-assign IPs to all node types
resource_registry:
  OS::TripleO::Controller::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external_from_pool.yaml
  OS::TripleO::Controller::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml
  OS::TripleO::Controller::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml
  OS::TripleO::Controller::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_mgmt_from_pool.yaml
  OS::TripleO::Controller::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant_from_pool.yaml

  OS::TripleO::Compute::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external_from_pool.yaml
  OS::TripleO::Compute::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml
  OS::TripleO::Compute::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml
  OS::TripleO::Compute::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Compute::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant_from_pool.yaml

  OS::TripleO::ComputeHCI::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external_from_pool.yaml
  OS::TripleO::ComputeHCI::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml
  OS::TripleO::ComputeHCI::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml
  OS::TripleO::ComputeHCI::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_mgmt_from_pool.yaml
  OS::TripleO::ComputeHCI::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant_from_pool.yaml

parameter_defaults:
  ControllerIPs:
    external:
    - 192.168.1.21
    - 192.168.1.22
    - 192.168.1.23
    internal_api:
    - 172.16.4.21
    - 172.16.4.22
    - 172.16.4.23
    storage:
    - 172.16.6.21
    - 172.16.6.22
    - 172.16.6.23
    storage_mgmt:
    - 172.16.7.21
    - 172.16.7.22
    - 172.16.7.23
    tenant:
    - 172.16.3.21
    - 172.16.3.22
    - 172.16.3.23
    ctlplane:
    - 10.1.1.21
    - 10.1.1.22
    - 10.1.1.23
  ComputeHCIIPs:
    external:
    - 192.168.1.28
    - 192.168.1.29
    - 192.168.1.30
    - 192.168.1.31
    internal_api:
    - 172.16.4.28
    - 172.16.4.29
    - 172.16.4.30
    - 172.16.4.31
    storage:
    - 172.16.6.28
    - 172.16.6.29
    - 172.16.6.30
    - 172.16.6.31
    storage_mgmt:
    - 172.16.7.28
    - 172.16.7.29
    - 172.16.7.30
    - 172.16.7.31
    tenant:
    - 172.16.3.28
    - 172.16.3.29
    - 172.16.3.30
    - 172.16.3.31
    ctlplane:
    - 10.1.1.28
    - 10.1.1.29
    - 10.1.1.30
    - 10.1.1.31
  ComputeIPs:
    external:
    - 192.168.1.24
    internal_api:
    - 172.16.4.24
    storage:
    - 172.16.6.24
    tenant:
    - 172.16.3.24
    ctlplane:
    - 10.1.1.24


(undercloud) [stack@director14 ~]$ openstack server list
+--------------------------------------+-------------+--------+---------------------+----------------+-----------+
| ID                                   | Name        | Status | Networks            | Image          | Flavor    |
+--------------------------------------+-------------+--------+---------------------+----------------+-----------+
| aae5ff37-9ebb-473d-b27f-ed7ab6f7c63b | hci1        | ACTIVE | ctlplane=10.1.1.28  | overcloud-full | baremetal |
| 52807bbc-09d2-4495-a290-a53c51462f88 | controller1 | ACTIVE | ctlplane=10.1.1.21  | overcloud-full | baremetal |
| cc2b682d-d066-41cc-915b-49341231eb2c | controller2 | ACTIVE | ctlplane=10.1.1.22  | overcloud-full | baremetal |
| bacb3ce0-dab3-49b1-9835-784eb5821f0a | controller3 | ACTIVE | ctlplane=10.1.1.23  | overcloud-full | baremetal |
| 81193cff-0265-4ac1-b4fe-018128225dd9 | compute1    | ACTIVE | ctlplane=10.1.1.114 | overcloud-full | baremetal |
| 375cc59d-07a5-49dd-94ce-44c0895351c1 | hci4        | ACTIVE | ctlplane=10.1.1.31  | overcloud-full | baremetal |
| 2365a268-62bd-4390-b71e-7db34685b222 | hci2        | ACTIVE | ctlplane=10.1.1.29  | overcloud-full | baremetal |
| c24b6f41-0401-43aa-9bb2-e9c9018c7cf9 | hci3        | ACTIVE | ctlplane=10.1.1.30  | overcloud-full | baremetal |
+--------------------------------------+-------------+--------+---------------------+----------------+-----------+

undercloud.conf
[DEFAULT]
local_ip = 10.1.1.2/24
undercloud_public_vip = 192.168.1.108
undercloud_admin_vip = 10.1.1.3
local_interface = eth0
masquerade_network = 10.1.1.0/24
dhcp_start = 10.1.1.100
dhcp_end = 10.1.1.150
network_cidr = 10.1.1.0/24
network_gateway = 10.1.1.2
inspection_iprange = 10.1.1.151,10.1.1.200
undercloud_debug = false
clean_nodes = true
overcloud_domain_name = lab.lan
undercloud_service_certificate = /etc/pki/instack-certs/undercloud.pem

Comment 3 Dan Sneddon 2019-02-13 22:15:52 UTC
(In reply to Ken Holden from comment #1)
> experienced this too.  deployed with 3 roles (Controller, ComputeHci, and
> Compute).  all controllers and HCI nodes deployed with static ctlplane IP,
> but the Compute node grabbed it's ctlplane IP from the DHCP pool in
> director.  

Could you please share the deployment command with included environment files? I'd like to see which environments are being included and in what order. Also, what is the name of the environment file where you define the "OS::TripleO::Compute::Ports::*"? What I suspect may have happened is that "OS::TripleO::Compute::Ports::*" are defined in your environment file, but there may be another environment file which is being included later which overwrites that value with the default of <network>.yaml.

In general, we advise including the network environment files last on the command line.

Thanks for including some more info. In the mean time, I'm looking through the OSP 14 codebase to see if there is a conflict anywhere.

Comment 5 Ken Savich 2019-02-14 15:07:16 UTC
So here's my deploy command from my initial bug report:

openstack overcloud deploy --templates --stack configtest \
  -r ~/templates/roles_data.yaml \
  -n ~/templates/network_data.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e ~/templates/network.yaml \
  -e ~/templates/node-info.yaml \
  -e ~/templates/storage-environment.yaml \
  -e ~/templates/ips-from-pool-ctlplane.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
  --ntp-server pool.ntp.org

I've gone ahead and redeployed, with the following deploy command:

openstack overcloud deploy --templates --stack configtest \
  -r ~/templates/roles_data.yaml \
  -e ~/templates/node-info.yaml \
  -e ~/templates/storage-environment.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e ~/templates/ips-from-pool-ctlplane.yaml \
  -n ~/templates/network_data.yaml \
  -e ~/templates/network.yaml \
  --ntp-server pool.ntp.org

Here's what my ips-from-pool-ctlplane.yaml looks like:

parameter_defaults:
  ControllerIPs:
    ctlplane:
    - 10.1.1.10
    - 10.1.1.11
    - 10.1.1.12
  ComputeIPs:
    ctlplane:
    - 10.1.1.13

Here are the results of the re-deploy:

(undercloud) [stack@director ~]$ openstack server list
+--------------------------------------+-------------------------+--------+--------------------+----------------+---------+
| ID                                   | Name                    | Status | Networks           | Image          | Flavor  |
+--------------------------------------+-------------------------+--------+--------------------+----------------+---------+
| af7ecf75-0dd2-403e-b4cd-f57a559ac033 | configtest-controller-2 | ACTIVE | ctlplane=10.1.1.12 | overcloud-full | control |
| af35ad55-ed52-42a8-bb04-6f23f05ded7c | configtest-controller-1 | ACTIVE | ctlplane=10.1.1.11 | overcloud-full | control |
| 635601c0-aadf-4f3e-bcee-91a6d772c8c1 | configtest-controller-0 | ACTIVE | ctlplane=10.1.1.10 | overcloud-full | control |
| 1e299f87-23ed-4b45-ac14-5122eef8f3ce | configtest-compute-0    | ACTIVE | ctlplane=10.1.1.31 | overcloud-full | compute |
+--------------------------------------+-------------------------+--------+--------------------+----------------+---------+

Note, my compute node is ip'd as 31 instead of 13

To answer your question regarding OS::TripleO::Compute::Ports::* - I only have this specified here:

(undercloud) [stack@director templates]$ grep -r OS::TripleO::Compute::Ports
nic-configs/README.md:    OS::TripleO::Compute::Ports::ManagementPort: ../network/ports/management.yaml

So, in effect, I'm not defining it at all.

Thanks Dan

Comment 6 Harald Jensås 2019-02-14 18:49:39 UTC
This is because the default Compute role have: 

  deprecated_param_ips: 'NovaComputeIPs'

See: https://github.com/openstack/tripleo-heat-templates/blob/0e68a0e30f5215e330ff386650a4d7d15742260b/roles/Compute.yaml#L27

 You must not have this fix: https://review.openstack.org/622349 ?

Change the `ips-from-pool-ctlplane.yaml` to:

parameter_defaults:
  ControllerIPs:
    ctlplane:
    - 10.1.1.10
    - 10.1.1.11
    - 10.1.1.12
  NovaComputeIPs:
    ctlplane:
    - 10.1.1.13

 ... or ... Remove the """ deprecated_param_ips: 'NovaComputeIPs' """ line from the role defenition.

Comment 7 Ken Savich 2019-02-14 19:57:27 UTC
Testing - 

One thing to note, I auto-generated the role definition from the z-stream templates. Am I correct in assuming then, that this would still be a bug -- because, according the documentation, the definition should be <role_name>IPs?

Comment 8 Harald Jensås 2019-02-14 20:27:03 UTC
(In reply to Ken Savich from comment #7)
> Testing - 
> 
> One thing to note, I auto-generated the role definition from the z-stream
> templates. Am I correct in assuming then, that this would still be a bug --
> because, according the documentation, the definition should be
> <role_name>IPs?

The role definitions does set the 'deprecated_param_ips' parameter. 

Unless you have this fix[1] , you would have to use what is set in deprecated_param_ips. IF you do have this fix, and it still does not work. Then we need to revisit this, i.e in that case what we did to close this issue does not work as intended.

[1] https://review.openstack.org/622349

Comment 9 Ken Savich 2019-02-14 20:36:26 UTC
Tested working, removing:

deprecated_param_ips: 'NovaComputeIPs

from my roles definition - 

(undercloud) [stack@director ~]$ openstack server list
+--------------------------------------+-------------------------+--------+--------------------+----------------+---------+
| ID                                   | Name                    | Status | Networks           | Image          | Flavor  |
+--------------------------------------+-------------------------+--------+--------------------+----------------+---------+
| bae2cc9f-5947-4d81-95b6-6c863e783f9c | configtest-controller-0 | ACTIVE | ctlplane=10.1.1.10 | overcloud-full | control |
| 13d561f9-e11d-4476-8b34-839cd535a919 | configtest-controller-2 | ACTIVE | ctlplane=10.1.1.12 | overcloud-full | control |
| 7824dfe6-2979-4346-bf0c-d781988f8033 | configtest-compute-0    | ACTIVE | ctlplane=10.1.1.13 | overcloud-full | compute |
| 2c42bf80-20f0-4cef-8d1d-99185cac8f42 | configtest-controller-1 | ACTIVE | ctlplane=10.1.1.11 | overcloud-full | control |
+--------------------------------------+-------------------------+--------+--------------------+----------------+---------+

Comment 10 Harald Jensås 2019-02-14 21:07:34 UTC

*** This bug has been marked as a duplicate of bug 1653684 ***