Deployment with composable roles and networks fails with: "2019-02-20 23:46:15Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller tenant1_ip_address) is incorrect."
DescriptionAlexander Chuzhoy
2019-02-21 14:08:15 UTC
Deployment with composable roles and networks fails with: "2019-02-20 23:46:15Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller tenant1_ip_address) is incorrect."
Environment:
openstack-tripleo-heat-templates-9.2.1-0.20190119154856.fe11ade.el7ost.noarch
instack-undercloud-9.4.1-0.20180928005746.15cda5a.el7ost.noarch
Steps to reproduce:
Attempt to deploy OC with composable networks and roles - the used examples are below:
Deployment command:
openstack overcloud deploy --templates \
--libvirt-type kvm \
-e /home/stack/templates/nodes_data.yaml \
-r /home/stack/templates/roles_data.yaml \
-n /home/stack/templates/network_data.yaml \
-e /home/stack/templates/extraconfig.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
-e /home/stack/virt/internal.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/templates/ips-from-pool-all.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
-e /home/stack/virt/enable-tls.yaml \
-e /home/stack/virt/public_vip.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/containers-prepare-parameter.yaml
Relevant part of roles yaml:
- name: Controller
description: |
Controller role that has all the controler services loaded and handles
Database, Messaging and Network functions.
CountDefault: 1
tags:
- primary
- controller
networks:
- External
- InternalApi
- Storage
- StorageMgmt
- Tenant
- Composable1
- Composable2
- name: Compute1
description: |
Basic Compute Node role
CountDefault: 1
networks:
- InternalApi1
- Tenant1
- Storage1
- name: Compute2
description: |
Basic Compute Node role
CountDefault: 1
networks:
- InternalApi2
- Tenant2
- Storage2
- name: CephStorage1
description: |
Ceph OSD Storage node role
networks:
- Storage1
- StorageMgmt1
Relevant part from network yaml:
- name: External
vip: true
name_lower: external
ip_subnet: '10.0.0.0/24'
allocation_pools: [{'start': '10.0.0.4', 'end': '10.0.0.250'}]
gateway_ip: '10.0.0.1'
ipv6_subnet: '2001:db8:fd00:1000::/64'
ipv6_allocation_pools: [{'start': '2001:db8:fd00:1000::10', 'end': '2001:db8:fd00:1000:ffff:ffff:ffff:fffe'}]
gateway_ipv6: '2001:db8:fd00:1000::1'
- name: InternalApi
name_lower: internal_api
vip: true
ip_subnet: '172.16.2.0/24'
allocation_pools: [{'start': '172.16.2.4', 'end': '172.16.2.250'}]
ipv6_subnet: 'fd00:fd00:fd00:2000::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2000::10', 'end': 'fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe'}]
- name: Storage
vip: true
name_lower: storage
ip_subnet: '172.16.1.0/24'
allocation_pools: [{'start': '172.16.1.4', 'end': '172.16.1.250'}]
ipv6_subnet: 'fd00:fd00:fd00:3000::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:3000::10', 'end': 'fd00:fd00:fd00:3000:ffff:ffff:ffff:fffe'}]
- name: StorageMgmt
name_lower: storage_mgmt
vip: true
ip_subnet: '172.16.3.0/24'
allocation_pools: [{'start': '172.16.3.4', 'end': '172.16.3.250'}]
ipv6_subnet: 'fd00:fd00:fd00:4000::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:4000::10', 'end': 'fd00:fd00:fd00:4000:ffff:ffff:ffff:fffe'}]
- name: Tenant
vip: false # Tenant network does not use VIPs
name_lower: tenant
ip_subnet: '172.16.0.0/24'
allocation_pools: [{'start': '172.16.0.4', 'end': '172.16.0.250'}]
ipv6_subnet: 'fd00:fd00:fd00:5000::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:5000::10', 'end': 'fd00:fd00:fd00:5000:ffff:ffff:ffff:fffe'}]
- name: Management
# Management network is enabled by default for backwards-compatibility, but
# is not included in any roles by default. Add to role definitions to use.
enabled: true
vip: false # Management network does not use VIPs
name_lower: management
ip_subnet: '10.0.1.0/24'
allocation_pools: [{'start': '10.0.1.4', 'end': '10.0.1.250'}]
ipv6_subnet: 'fd00:fd00:fd00:6000::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:6000::10', 'end': 'fd00:fd00:fd00:6000:ffff:ffff:ffff:fffe'}]
- name: Tenant1
vip: false # Tenant network does not use VIPs
name_lower: tenant1
ip_subnet: '172.16.11.0/24'
allocation_pools: [{'start': '172.16.11.4', 'end': '172.16.11.250'}]
ipv6_subnet: 'fd00:fd00:fd00:5001::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:5001::10', 'end': 'fd00:fd00:fd00:5001:ffff:ffff:ffff:fffe'}]
- name: Tenant2
vip: false # Tenant network does not use VIPs
name_lower: tenant2
ip_subnet: '172.16.12.0/24'
allocation_pools: [{'start': '172.16.12.4', 'end': '172.16.12.250'}]
ipv6_subnet: 'fd00:fd00:fd00:5002::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:5002::10', 'end': 'fd00:fd00:fd00:5002:ffff:ffff:ffff:fffe'}]
- name: Tenant3
vip: false # Tenant network does not use VIPs
name_lower: tenant3
ip_subnet: '172.16.13.0/24'
allocation_pools: [{'start': '172.16.13.4', 'end': '172.16.13.250'}]
ipv6_subnet: 'fd00:fd00:fd00:5003::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:5003::10', 'end': 'fd00:fd00:fd00:5003:ffff:ffff:ffff:fffe'}]
- name: StorageMgmt1
name_lower: storage_mgmt1
vip: true
ip_subnet: '172.16.21.0/24'
allocation_pools: [{'start': '172.16.21.4', 'end': '172.16.21.250'}]
ipv6_subnet: 'fd00:fd00:fd00:4001::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:4001::10', 'end': 'fd00:fd00:fd00:4001:ffff:ffff:ffff:fffe'}]
- name: StorageMgmt2
name_lower: storage_mgmt2
vip: true
ip_subnet: '172.16.22.0/24'
allocation_pools: [{'start': '172.16.22.4', 'end': '172.16.22.250'}]
ipv6_subnet: 'fd00:fd00:fd00:4002::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:4002::10', 'end': 'fd00:fd00:fd00:4002:ffff:ffff:ffff:fffe'}]
- name: StorageMgmt3
name_lower: storage_mgmt3
vip: true
ip_subnet: '172.16.23.0/24'
allocation_pools: [{'start': '172.16.23.4', 'end': '172.16.23.250'}]
ipv6_subnet: 'fd00:fd00:fd00:4003::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:4003::10', 'end': 'fd00:fd00:fd00:4003:ffff:ffff:ffff:fffe'}]
- name: Storage1
vip: true
name_lower: storage1
ip_subnet: '172.16.31.0/24'
allocation_pools: [{'start': '172.16.31.4', 'end': '172.16.31.250'}]
ipv6_subnet: 'fd00:fd00:fd00:3001::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:3001::10', 'end': 'fd00:fd00:fd00:3001:ffff:ffff:ffff:fffe'}]
- name: Storage2
vip: true
name_lower: storage2
ip_subnet: '172.16.32.0/24'
allocation_pools: [{'start': '172.16.32.4', 'end': '172.16.32.250'}]
ipv6_subnet: 'fd00:fd00:fd00:3002::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:3002::10', 'end': 'fd00:fd00:fd00:3002:ffff:ffff:ffff:fffe'}]
- name: Storage3
vip: true
name_lower: storage3
ip_subnet: '172.16.33.0/24'
allocation_pools: [{'start': '172.16.33.4', 'end': '172.16.33.250'}]
ipv6_subnet: 'fd00:fd00:fd00:3003::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:3003::10', 'end': 'fd00:fd00:fd00:3003:ffff:ffff:ffff:fffe'}]
- name: InternalApi1
name_lower: internal_api1
vip: true
ip_subnet: '172.16.41.0/24'
allocation_pools: [{'start': '172.16.41.4', 'end': '172.16.41.250'}]
ipv6_subnet: 'fd00:fd00:fd00:2001::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2001::10', 'end': 'fd00:fd00:fd00:2001:ffff:ffff:ffff:fffe'}]
- name: InternalApi2
name_lower: internal_api2
vip: true
ip_subnet: '172.16.42.0/24'
allocation_pools: [{'start': '172.16.42.4', 'end': '172.16.42.250'}]
ipv6_subnet: 'fd00:fd00:fd00:2002::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2002::10', 'end': 'fd00:fd00:fd00:2002:ffff:ffff:ffff:fffe'}]
- name: InternalApi3
name_lower: internal_api3
vip: true
ip_subnet: '172.16.43.0/24'
allocation_pools: [{'start': '172.16.43.4', 'end': '172.16.43.250'}]
ipv6_subnet: 'fd00:fd00:fd00:2003::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2003::10', 'end': 'fd00:fd00:fd00:2003:ffff:ffff:ffff:fffe'}]
- name: Composable1
name_lower: composable1
vip: true
ip_subnet: '172.16.44.0/24'
allocation_pools: [{'start': '172.16.44.4', 'end': '172.16.44.250'}]
ipv6_subnet: 'fd00:fd00:fd00:2004::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2004::10', 'end': 'fd00:fd00:fd00:2004:ffff:ffff:ffff:fffe'}]
- name: Composable2
name_lower: composable2
vip: true
ipv6: true
ip_subnet: '172.16.45.0/24'
allocation_pools: [{'start': '172.16.45.4', 'end': '172.16.45.250'}]
ipv6_subnet: 'fd00:fd00:fd00:2005::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2005::10', 'end': 'fd00:fd00:fd00:2005:ffff:ffff:ffff:fffe'}]
Part of nic config yaml:
parameters:
InternalApiInterfaceDefaultRoute: # Override this via parameter_defaults
description: Default route for the specific network.
type: string
TenantInterfaceDefaultRoute: # Override this via parameter_defaults
description: Default route for the specific network.
type: string
StorageInterfaceDefaultRoute: # Override this via parameter_defaults
description: Default route for the specific network.
type: string
StorageMgmtInterfaceDefaultRoute: # Override this via parameter_defaults
description: Default route for the specific network.
type: string
InternalApi1NetworkVlanID:
default: 21
description: Vlan ID for the internal_api network traffic.
type: number
InternalApi2NetworkVlanID:
default: 22
description: Vlan ID for the internal_api network traffic.
type: number
InternalApi3NetworkVlanID:
default: 23
description: Vlan ID for the internal_api network traffic.
type: number
Storage1NetworkVlanID:
default: 31
description: Vlan ID for the storage network traffic.
type: number
Storage2NetworkVlanID:
default: 32
description: Vlan ID for the storage network traffic.
type: number
Storage3NetworkVlanID:
default: 33
description: Vlan ID for the storage network traffic.
type: number
StorageMgmt1NetworkVlanID:
default: 41
description: Vlan ID for the storage mgmt network traffic.
type: number
StorageMgmt2NetworkVlanID:
default: 42
description: Vlan ID for the storage mgmt network traffic.
type: number
StorageMgmt3NetworkVlanID:
default: 43
description: Vlan ID for the storage mgmt network traffic.
type: number
Composable1NetworkVlanID:
default: 54
description: Vlan ID for the tenant network traffic.
type: number
Composable2NetworkVlanID:
default: 55
description: Vlan ID for the tenant network traffic.
type: number
InternalApi1IpSubnet:
default: ''
description: IP address/subnet on the internal API network
type: string
InternalApi2IpSubnet:
default: ''
description: IP address/subnet on the internal API network
type: string
InternalApi3IpSubnet:
default: ''
description: IP address/subnet on the internal API network
type: string
Storage1IpSubnet:
default: ''
description: IP address/subnet on the storage network
type: string
Storage2IpSubnet:
default: ''
description: IP address/subnet on the storage network
type: string
Storage3IpSubnet:
default: ''
description: IP address/subnet on the storage network
type: string
StorageMgmt1IpSubnet:
default: ''
description: IP address/subnet on the storage mgmt network
type: string
StorageMgmt2IpSubnet:
default: ''
description: IP address/subnet on the storage mgmt network
type: string
StorageMgmt3IpSubnet:
default: ''
description: IP address/subnet on the storage mgmt network
type: string
Composable1IpSubnet:
default: ''
description: IP address/subnet on the tenant network
type: string
Composable2IpSubnet:
default: ''
description: IP address/subnet on the tenant network
type: string
ControlPlaneIp:
default: ''
description: IP address/subnet on the ctlplane network
type: string
ExternalIpSubnet:
default: ''
description: IP address/subnet on the external network
type: string
InternalApiIpSubnet:
default: ''
description: IP address/subnet on the internal API network
type: string
StorageIpSubnet:
default: ''
description: IP address/subnet on the storage network
type: string
StorageMgmtIpSubnet:
default: ''
description: IP address/subnet on the storage mgmt network
type: string
TenantIpSubnet:
default: ''
description: IP address/subnet on the tenant network
type: string
ManagementIpSubnet: # Only populated when including environments/network-management.yaml
default: ''
description: IP address/subnet on the management network
type: string
ExternalNetworkVlanID:
default: 10
description: Vlan ID for the external network traffic.
type: number
InternalApiNetworkVlanID:
default: 20
description: Vlan ID for the internal_api network traffic.
type: number
StorageNetworkVlanID:
default: 30
description: Vlan ID for the storage network traffic.
type: number
StorageMgmtNetworkVlanID:
default: 40
description: Vlan ID for the storage mgmt network traffic.
type: number
TenantNetworkVlanID:
default: 50
description: Vlan ID for the tenant network traffic.
type: number
ExternalInterfaceDefaultRoute:
default: 10.0.0.1
description: default route for the external network
type: string
ControlPlaneSubnetCidr: # Override this via parameter_defaults
default: '24'
description: The subnet CIDR of the control plane network.
type: string
ControlPlaneDefaultRoute: # Override this via parameter_defaults
description: The subnet CIDR of the control plane network.
type: string
DnsServers: # Override this via parameter_defaults
default: []
description: A list of DNS servers (2 max for some implementations) that will be added to resolv.conf.
type: json
EC2MetadataIp: # Override this via parameter_defaults
description: The IP address of the EC2 metadata server.
type: string
Composable1InterfaceRoutes:
default: []
description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
Composable2InterfaceRoutes:
default: []
description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
StorageInterfaceRoutes:
default: []
description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
StorageMgmtInterfaceRoutes:
default: []
description: 'Routes for the storage_mgmt network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
InternalApiInterfaceRoutes:
default: []
description: 'Routes for the internal_api network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
TenantInterfaceRoutes:
default: []
description: 'Routes for the tenant network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
ExternalInterfaceRoutes:
default: []
description: 'Routes for the external network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
ManagementInterfaceRoutes:
default: []
description: 'Routes for the management network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
Storage1InterfaceRoutes:
default: []
description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
StorageMgmt1InterfaceRoutes:
default: []
description: 'Routes for the storage_mgmt network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
InternalApi1InterfaceRoutes:
default: []
description: 'Routes for the internal_api network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
External1InterfaceRoutes:
default: []
description: 'Routes for the external network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
Management1InterfaceRoutes:
default: []
description: 'Routes for the management network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
Storage2InterfaceRoutes:
default: []
description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
StorageMgmt2InterfaceRoutes:
default: []
description: 'Routes for the storage_mgmt network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
InternalApi2InterfaceRoutes:
default: []
description: 'Routes for the internal_api network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
External2InterfaceRoutes:
default: []
description: 'Routes for the external network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
Management2InterfaceRoutes:
default: []
description: 'Routes for the management network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
Storage3InterfaceRoutes:
default: []
description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
StorageMgmt3InterfaceRoutes:
default: []
description: 'Routes for the storage_mgmt network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
InternalApi3InterfaceRoutes:
default: []
description: 'Routes for the internal_api network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
External3InterfaceRoutes:
default: []
description: 'Routes for the external network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
Management3InterfaceRoutes:
default: []
description: 'Routes for the management network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}]
Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. '
type: json
Result:
The overcloud deployment fails with:
"2019-02-20 23:46:15Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller tenant1_ip_address) is incorrect."
Tried to remove tenant1 from controller's yaml to see where it takes us. Now there's a new error upon failure:
2019-02-21 03:01:28Z [overcloud.Controller.1]: CREATE_COMPLETE state changed
2019-02-21 03:01:28Z [overcloud.Controller]: UPDATE_COMPLETE Stack UPDATE completed successfully
2019-02-21 03:01:28Z [overcloud.Controller]: CREATE_COMPLETE state changed
2019-02-21 03:01:30Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller tenant1_ip_address) is incorrect.
2019-02-21 03:01:31Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller storage_mgmt3_ip_address) is incorrect.
2019-02-21 03:01:31Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller storage_mgmt2_ip_address) is incorrect.
Stack overcloud/1842300f-d20a-4af8-bbb5-3ba7791fafa5 CREATE_FAILED
Heat Stack create failed.
Heat Stack create failed.
This happens pretty early - before the os-net-config (the OC nodes booted, but the proper addresses wasn't set on nodes).
Comment 4Alexander Chuzhoy
2019-02-22 17:52:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2019:0446
Deployment with composable roles and networks fails with: "2019-02-20 23:46:15Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller tenant1_ip_address) is incorrect." Environment: openstack-tripleo-heat-templates-9.2.1-0.20190119154856.fe11ade.el7ost.noarch instack-undercloud-9.4.1-0.20180928005746.15cda5a.el7ost.noarch Steps to reproduce: Attempt to deploy OC with composable networks and roles - the used examples are below: Deployment command: openstack overcloud deploy --templates \ --libvirt-type kvm \ -e /home/stack/templates/nodes_data.yaml \ -r /home/stack/templates/roles_data.yaml \ -n /home/stack/templates/network_data.yaml \ -e /home/stack/templates/extraconfig.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /home/stack/virt/internal.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/templates/ips-from-pool-all.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \ -e /home/stack/virt/enable-tls.yaml \ -e /home/stack/virt/public_vip.yaml \ -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/containers-prepare-parameter.yaml Relevant part of roles yaml: - name: Controller description: | Controller role that has all the controler services loaded and handles Database, Messaging and Network functions. CountDefault: 1 tags: - primary - controller networks: - External - InternalApi - Storage - StorageMgmt - Tenant - Composable1 - Composable2 - name: Compute1 description: | Basic Compute Node role CountDefault: 1 networks: - InternalApi1 - Tenant1 - Storage1 - name: Compute2 description: | Basic Compute Node role CountDefault: 1 networks: - InternalApi2 - Tenant2 - Storage2 - name: CephStorage1 description: | Ceph OSD Storage node role networks: - Storage1 - StorageMgmt1 Relevant part from network yaml: - name: External vip: true name_lower: external ip_subnet: '10.0.0.0/24' allocation_pools: [{'start': '10.0.0.4', 'end': '10.0.0.250'}] gateway_ip: '10.0.0.1' ipv6_subnet: '2001:db8:fd00:1000::/64' ipv6_allocation_pools: [{'start': '2001:db8:fd00:1000::10', 'end': '2001:db8:fd00:1000:ffff:ffff:ffff:fffe'}] gateway_ipv6: '2001:db8:fd00:1000::1' - name: InternalApi name_lower: internal_api vip: true ip_subnet: '172.16.2.0/24' allocation_pools: [{'start': '172.16.2.4', 'end': '172.16.2.250'}] ipv6_subnet: 'fd00:fd00:fd00:2000::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2000::10', 'end': 'fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe'}] - name: Storage vip: true name_lower: storage ip_subnet: '172.16.1.0/24' allocation_pools: [{'start': '172.16.1.4', 'end': '172.16.1.250'}] ipv6_subnet: 'fd00:fd00:fd00:3000::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:3000::10', 'end': 'fd00:fd00:fd00:3000:ffff:ffff:ffff:fffe'}] - name: StorageMgmt name_lower: storage_mgmt vip: true ip_subnet: '172.16.3.0/24' allocation_pools: [{'start': '172.16.3.4', 'end': '172.16.3.250'}] ipv6_subnet: 'fd00:fd00:fd00:4000::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:4000::10', 'end': 'fd00:fd00:fd00:4000:ffff:ffff:ffff:fffe'}] - name: Tenant vip: false # Tenant network does not use VIPs name_lower: tenant ip_subnet: '172.16.0.0/24' allocation_pools: [{'start': '172.16.0.4', 'end': '172.16.0.250'}] ipv6_subnet: 'fd00:fd00:fd00:5000::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:5000::10', 'end': 'fd00:fd00:fd00:5000:ffff:ffff:ffff:fffe'}] - name: Management # Management network is enabled by default for backwards-compatibility, but # is not included in any roles by default. Add to role definitions to use. enabled: true vip: false # Management network does not use VIPs name_lower: management ip_subnet: '10.0.1.0/24' allocation_pools: [{'start': '10.0.1.4', 'end': '10.0.1.250'}] ipv6_subnet: 'fd00:fd00:fd00:6000::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:6000::10', 'end': 'fd00:fd00:fd00:6000:ffff:ffff:ffff:fffe'}] - name: Tenant1 vip: false # Tenant network does not use VIPs name_lower: tenant1 ip_subnet: '172.16.11.0/24' allocation_pools: [{'start': '172.16.11.4', 'end': '172.16.11.250'}] ipv6_subnet: 'fd00:fd00:fd00:5001::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:5001::10', 'end': 'fd00:fd00:fd00:5001:ffff:ffff:ffff:fffe'}] - name: Tenant2 vip: false # Tenant network does not use VIPs name_lower: tenant2 ip_subnet: '172.16.12.0/24' allocation_pools: [{'start': '172.16.12.4', 'end': '172.16.12.250'}] ipv6_subnet: 'fd00:fd00:fd00:5002::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:5002::10', 'end': 'fd00:fd00:fd00:5002:ffff:ffff:ffff:fffe'}] - name: Tenant3 vip: false # Tenant network does not use VIPs name_lower: tenant3 ip_subnet: '172.16.13.0/24' allocation_pools: [{'start': '172.16.13.4', 'end': '172.16.13.250'}] ipv6_subnet: 'fd00:fd00:fd00:5003::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:5003::10', 'end': 'fd00:fd00:fd00:5003:ffff:ffff:ffff:fffe'}] - name: StorageMgmt1 name_lower: storage_mgmt1 vip: true ip_subnet: '172.16.21.0/24' allocation_pools: [{'start': '172.16.21.4', 'end': '172.16.21.250'}] ipv6_subnet: 'fd00:fd00:fd00:4001::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:4001::10', 'end': 'fd00:fd00:fd00:4001:ffff:ffff:ffff:fffe'}] - name: StorageMgmt2 name_lower: storage_mgmt2 vip: true ip_subnet: '172.16.22.0/24' allocation_pools: [{'start': '172.16.22.4', 'end': '172.16.22.250'}] ipv6_subnet: 'fd00:fd00:fd00:4002::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:4002::10', 'end': 'fd00:fd00:fd00:4002:ffff:ffff:ffff:fffe'}] - name: StorageMgmt3 name_lower: storage_mgmt3 vip: true ip_subnet: '172.16.23.0/24' allocation_pools: [{'start': '172.16.23.4', 'end': '172.16.23.250'}] ipv6_subnet: 'fd00:fd00:fd00:4003::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:4003::10', 'end': 'fd00:fd00:fd00:4003:ffff:ffff:ffff:fffe'}] - name: Storage1 vip: true name_lower: storage1 ip_subnet: '172.16.31.0/24' allocation_pools: [{'start': '172.16.31.4', 'end': '172.16.31.250'}] ipv6_subnet: 'fd00:fd00:fd00:3001::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:3001::10', 'end': 'fd00:fd00:fd00:3001:ffff:ffff:ffff:fffe'}] - name: Storage2 vip: true name_lower: storage2 ip_subnet: '172.16.32.0/24' allocation_pools: [{'start': '172.16.32.4', 'end': '172.16.32.250'}] ipv6_subnet: 'fd00:fd00:fd00:3002::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:3002::10', 'end': 'fd00:fd00:fd00:3002:ffff:ffff:ffff:fffe'}] - name: Storage3 vip: true name_lower: storage3 ip_subnet: '172.16.33.0/24' allocation_pools: [{'start': '172.16.33.4', 'end': '172.16.33.250'}] ipv6_subnet: 'fd00:fd00:fd00:3003::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:3003::10', 'end': 'fd00:fd00:fd00:3003:ffff:ffff:ffff:fffe'}] - name: InternalApi1 name_lower: internal_api1 vip: true ip_subnet: '172.16.41.0/24' allocation_pools: [{'start': '172.16.41.4', 'end': '172.16.41.250'}] ipv6_subnet: 'fd00:fd00:fd00:2001::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2001::10', 'end': 'fd00:fd00:fd00:2001:ffff:ffff:ffff:fffe'}] - name: InternalApi2 name_lower: internal_api2 vip: true ip_subnet: '172.16.42.0/24' allocation_pools: [{'start': '172.16.42.4', 'end': '172.16.42.250'}] ipv6_subnet: 'fd00:fd00:fd00:2002::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2002::10', 'end': 'fd00:fd00:fd00:2002:ffff:ffff:ffff:fffe'}] - name: InternalApi3 name_lower: internal_api3 vip: true ip_subnet: '172.16.43.0/24' allocation_pools: [{'start': '172.16.43.4', 'end': '172.16.43.250'}] ipv6_subnet: 'fd00:fd00:fd00:2003::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2003::10', 'end': 'fd00:fd00:fd00:2003:ffff:ffff:ffff:fffe'}] - name: Composable1 name_lower: composable1 vip: true ip_subnet: '172.16.44.0/24' allocation_pools: [{'start': '172.16.44.4', 'end': '172.16.44.250'}] ipv6_subnet: 'fd00:fd00:fd00:2004::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2004::10', 'end': 'fd00:fd00:fd00:2004:ffff:ffff:ffff:fffe'}] - name: Composable2 name_lower: composable2 vip: true ipv6: true ip_subnet: '172.16.45.0/24' allocation_pools: [{'start': '172.16.45.4', 'end': '172.16.45.250'}] ipv6_subnet: 'fd00:fd00:fd00:2005::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2005::10', 'end': 'fd00:fd00:fd00:2005:ffff:ffff:ffff:fffe'}] Part of nic config yaml: parameters: InternalApiInterfaceDefaultRoute: # Override this via parameter_defaults description: Default route for the specific network. type: string TenantInterfaceDefaultRoute: # Override this via parameter_defaults description: Default route for the specific network. type: string StorageInterfaceDefaultRoute: # Override this via parameter_defaults description: Default route for the specific network. type: string StorageMgmtInterfaceDefaultRoute: # Override this via parameter_defaults description: Default route for the specific network. type: string InternalApi1NetworkVlanID: default: 21 description: Vlan ID for the internal_api network traffic. type: number InternalApi2NetworkVlanID: default: 22 description: Vlan ID for the internal_api network traffic. type: number InternalApi3NetworkVlanID: default: 23 description: Vlan ID for the internal_api network traffic. type: number Storage1NetworkVlanID: default: 31 description: Vlan ID for the storage network traffic. type: number Storage2NetworkVlanID: default: 32 description: Vlan ID for the storage network traffic. type: number Storage3NetworkVlanID: default: 33 description: Vlan ID for the storage network traffic. type: number StorageMgmt1NetworkVlanID: default: 41 description: Vlan ID for the storage mgmt network traffic. type: number StorageMgmt2NetworkVlanID: default: 42 description: Vlan ID for the storage mgmt network traffic. type: number StorageMgmt3NetworkVlanID: default: 43 description: Vlan ID for the storage mgmt network traffic. type: number Composable1NetworkVlanID: default: 54 description: Vlan ID for the tenant network traffic. type: number Composable2NetworkVlanID: default: 55 description: Vlan ID for the tenant network traffic. type: number InternalApi1IpSubnet: default: '' description: IP address/subnet on the internal API network type: string InternalApi2IpSubnet: default: '' description: IP address/subnet on the internal API network type: string InternalApi3IpSubnet: default: '' description: IP address/subnet on the internal API network type: string Storage1IpSubnet: default: '' description: IP address/subnet on the storage network type: string Storage2IpSubnet: default: '' description: IP address/subnet on the storage network type: string Storage3IpSubnet: default: '' description: IP address/subnet on the storage network type: string StorageMgmt1IpSubnet: default: '' description: IP address/subnet on the storage mgmt network type: string StorageMgmt2IpSubnet: default: '' description: IP address/subnet on the storage mgmt network type: string StorageMgmt3IpSubnet: default: '' description: IP address/subnet on the storage mgmt network type: string Composable1IpSubnet: default: '' description: IP address/subnet on the tenant network type: string Composable2IpSubnet: default: '' description: IP address/subnet on the tenant network type: string ControlPlaneIp: default: '' description: IP address/subnet on the ctlplane network type: string ExternalIpSubnet: default: '' description: IP address/subnet on the external network type: string InternalApiIpSubnet: default: '' description: IP address/subnet on the internal API network type: string StorageIpSubnet: default: '' description: IP address/subnet on the storage network type: string StorageMgmtIpSubnet: default: '' description: IP address/subnet on the storage mgmt network type: string TenantIpSubnet: default: '' description: IP address/subnet on the tenant network type: string ManagementIpSubnet: # Only populated when including environments/network-management.yaml default: '' description: IP address/subnet on the management network type: string ExternalNetworkVlanID: default: 10 description: Vlan ID for the external network traffic. type: number InternalApiNetworkVlanID: default: 20 description: Vlan ID for the internal_api network traffic. type: number StorageNetworkVlanID: default: 30 description: Vlan ID for the storage network traffic. type: number StorageMgmtNetworkVlanID: default: 40 description: Vlan ID for the storage mgmt network traffic. type: number TenantNetworkVlanID: default: 50 description: Vlan ID for the tenant network traffic. type: number ExternalInterfaceDefaultRoute: default: 10.0.0.1 description: default route for the external network type: string ControlPlaneSubnetCidr: # Override this via parameter_defaults default: '24' description: The subnet CIDR of the control plane network. type: string ControlPlaneDefaultRoute: # Override this via parameter_defaults description: The subnet CIDR of the control plane network. type: string DnsServers: # Override this via parameter_defaults default: [] description: A list of DNS servers (2 max for some implementations) that will be added to resolv.conf. type: json EC2MetadataIp: # Override this via parameter_defaults description: The IP address of the EC2 metadata server. type: string Composable1InterfaceRoutes: default: [] description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Composable2InterfaceRoutes: default: [] description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json StorageInterfaceRoutes: default: [] description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json StorageMgmtInterfaceRoutes: default: [] description: 'Routes for the storage_mgmt network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json InternalApiInterfaceRoutes: default: [] description: 'Routes for the internal_api network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json TenantInterfaceRoutes: default: [] description: 'Routes for the tenant network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json ExternalInterfaceRoutes: default: [] description: 'Routes for the external network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json ManagementInterfaceRoutes: default: [] description: 'Routes for the management network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Storage1InterfaceRoutes: default: [] description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json StorageMgmt1InterfaceRoutes: default: [] description: 'Routes for the storage_mgmt network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json InternalApi1InterfaceRoutes: default: [] description: 'Routes for the internal_api network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json External1InterfaceRoutes: default: [] description: 'Routes for the external network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Management1InterfaceRoutes: default: [] description: 'Routes for the management network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Storage2InterfaceRoutes: default: [] description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json StorageMgmt2InterfaceRoutes: default: [] description: 'Routes for the storage_mgmt network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json InternalApi2InterfaceRoutes: default: [] description: 'Routes for the internal_api network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json External2InterfaceRoutes: default: [] description: 'Routes for the external network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Management2InterfaceRoutes: default: [] description: 'Routes for the management network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Storage3InterfaceRoutes: default: [] description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json StorageMgmt3InterfaceRoutes: default: [] description: 'Routes for the storage_mgmt network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json InternalApi3InterfaceRoutes: default: [] description: 'Routes for the internal_api network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json External3InterfaceRoutes: default: [] description: 'Routes for the external network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Management3InterfaceRoutes: default: [] description: 'Routes for the management network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Result: The overcloud deployment fails with: "2019-02-20 23:46:15Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller tenant1_ip_address) is incorrect." Tried to remove tenant1 from controller's yaml to see where it takes us. Now there's a new error upon failure: 2019-02-21 03:01:28Z [overcloud.Controller.1]: CREATE_COMPLETE state changed 2019-02-21 03:01:28Z [overcloud.Controller]: UPDATE_COMPLETE Stack UPDATE completed successfully 2019-02-21 03:01:28Z [overcloud.Controller]: CREATE_COMPLETE state changed 2019-02-21 03:01:30Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller tenant1_ip_address) is incorrect. 2019-02-21 03:01:31Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller storage_mgmt3_ip_address) is incorrect. 2019-02-21 03:01:31Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller storage_mgmt2_ip_address) is incorrect. Stack overcloud/1842300f-d20a-4af8-bbb5-3ba7791fafa5 CREATE_FAILED Heat Stack create failed. Heat Stack create failed. This happens pretty early - before the os-net-config (the OC nodes booted, but the proper addresses wasn't set on nodes).