Deployment with composable roles and networks fails with: "2019-02-20 23:46:15Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller tenant1_ip_address) is incorrect." Environment: openstack-tripleo-heat-templates-9.2.1-0.20190119154856.fe11ade.el7ost.noarch instack-undercloud-9.4.1-0.20180928005746.15cda5a.el7ost.noarch Steps to reproduce: Attempt to deploy OC with composable networks and roles - the used examples are below: Deployment command: openstack overcloud deploy --templates \ --libvirt-type kvm \ -e /home/stack/templates/nodes_data.yaml \ -r /home/stack/templates/roles_data.yaml \ -n /home/stack/templates/network_data.yaml \ -e /home/stack/templates/extraconfig.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /home/stack/virt/internal.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/templates/ips-from-pool-all.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \ -e /home/stack/virt/enable-tls.yaml \ -e /home/stack/virt/public_vip.yaml \ -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/containers-prepare-parameter.yaml Relevant part of roles yaml: - name: Controller description: | Controller role that has all the controler services loaded and handles Database, Messaging and Network functions. CountDefault: 1 tags: - primary - controller networks: - External - InternalApi - Storage - StorageMgmt - Tenant - Composable1 - Composable2 - name: Compute1 description: | Basic Compute Node role CountDefault: 1 networks: - InternalApi1 - Tenant1 - Storage1 - name: Compute2 description: | Basic Compute Node role CountDefault: 1 networks: - InternalApi2 - Tenant2 - Storage2 - name: CephStorage1 description: | Ceph OSD Storage node role networks: - Storage1 - StorageMgmt1 Relevant part from network yaml: - name: External vip: true name_lower: external ip_subnet: '10.0.0.0/24' allocation_pools: [{'start': '10.0.0.4', 'end': '10.0.0.250'}] gateway_ip: '10.0.0.1' ipv6_subnet: '2001:db8:fd00:1000::/64' ipv6_allocation_pools: [{'start': '2001:db8:fd00:1000::10', 'end': '2001:db8:fd00:1000:ffff:ffff:ffff:fffe'}] gateway_ipv6: '2001:db8:fd00:1000::1' - name: InternalApi name_lower: internal_api vip: true ip_subnet: '172.16.2.0/24' allocation_pools: [{'start': '172.16.2.4', 'end': '172.16.2.250'}] ipv6_subnet: 'fd00:fd00:fd00:2000::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2000::10', 'end': 'fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe'}] - name: Storage vip: true name_lower: storage ip_subnet: '172.16.1.0/24' allocation_pools: [{'start': '172.16.1.4', 'end': '172.16.1.250'}] ipv6_subnet: 'fd00:fd00:fd00:3000::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:3000::10', 'end': 'fd00:fd00:fd00:3000:ffff:ffff:ffff:fffe'}] - name: StorageMgmt name_lower: storage_mgmt vip: true ip_subnet: '172.16.3.0/24' allocation_pools: [{'start': '172.16.3.4', 'end': '172.16.3.250'}] ipv6_subnet: 'fd00:fd00:fd00:4000::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:4000::10', 'end': 'fd00:fd00:fd00:4000:ffff:ffff:ffff:fffe'}] - name: Tenant vip: false # Tenant network does not use VIPs name_lower: tenant ip_subnet: '172.16.0.0/24' allocation_pools: [{'start': '172.16.0.4', 'end': '172.16.0.250'}] ipv6_subnet: 'fd00:fd00:fd00:5000::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:5000::10', 'end': 'fd00:fd00:fd00:5000:ffff:ffff:ffff:fffe'}] - name: Management # Management network is enabled by default for backwards-compatibility, but # is not included in any roles by default. Add to role definitions to use. enabled: true vip: false # Management network does not use VIPs name_lower: management ip_subnet: '10.0.1.0/24' allocation_pools: [{'start': '10.0.1.4', 'end': '10.0.1.250'}] ipv6_subnet: 'fd00:fd00:fd00:6000::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:6000::10', 'end': 'fd00:fd00:fd00:6000:ffff:ffff:ffff:fffe'}] - name: Tenant1 vip: false # Tenant network does not use VIPs name_lower: tenant1 ip_subnet: '172.16.11.0/24' allocation_pools: [{'start': '172.16.11.4', 'end': '172.16.11.250'}] ipv6_subnet: 'fd00:fd00:fd00:5001::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:5001::10', 'end': 'fd00:fd00:fd00:5001:ffff:ffff:ffff:fffe'}] - name: Tenant2 vip: false # Tenant network does not use VIPs name_lower: tenant2 ip_subnet: '172.16.12.0/24' allocation_pools: [{'start': '172.16.12.4', 'end': '172.16.12.250'}] ipv6_subnet: 'fd00:fd00:fd00:5002::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:5002::10', 'end': 'fd00:fd00:fd00:5002:ffff:ffff:ffff:fffe'}] - name: Tenant3 vip: false # Tenant network does not use VIPs name_lower: tenant3 ip_subnet: '172.16.13.0/24' allocation_pools: [{'start': '172.16.13.4', 'end': '172.16.13.250'}] ipv6_subnet: 'fd00:fd00:fd00:5003::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:5003::10', 'end': 'fd00:fd00:fd00:5003:ffff:ffff:ffff:fffe'}] - name: StorageMgmt1 name_lower: storage_mgmt1 vip: true ip_subnet: '172.16.21.0/24' allocation_pools: [{'start': '172.16.21.4', 'end': '172.16.21.250'}] ipv6_subnet: 'fd00:fd00:fd00:4001::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:4001::10', 'end': 'fd00:fd00:fd00:4001:ffff:ffff:ffff:fffe'}] - name: StorageMgmt2 name_lower: storage_mgmt2 vip: true ip_subnet: '172.16.22.0/24' allocation_pools: [{'start': '172.16.22.4', 'end': '172.16.22.250'}] ipv6_subnet: 'fd00:fd00:fd00:4002::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:4002::10', 'end': 'fd00:fd00:fd00:4002:ffff:ffff:ffff:fffe'}] - name: StorageMgmt3 name_lower: storage_mgmt3 vip: true ip_subnet: '172.16.23.0/24' allocation_pools: [{'start': '172.16.23.4', 'end': '172.16.23.250'}] ipv6_subnet: 'fd00:fd00:fd00:4003::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:4003::10', 'end': 'fd00:fd00:fd00:4003:ffff:ffff:ffff:fffe'}] - name: Storage1 vip: true name_lower: storage1 ip_subnet: '172.16.31.0/24' allocation_pools: [{'start': '172.16.31.4', 'end': '172.16.31.250'}] ipv6_subnet: 'fd00:fd00:fd00:3001::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:3001::10', 'end': 'fd00:fd00:fd00:3001:ffff:ffff:ffff:fffe'}] - name: Storage2 vip: true name_lower: storage2 ip_subnet: '172.16.32.0/24' allocation_pools: [{'start': '172.16.32.4', 'end': '172.16.32.250'}] ipv6_subnet: 'fd00:fd00:fd00:3002::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:3002::10', 'end': 'fd00:fd00:fd00:3002:ffff:ffff:ffff:fffe'}] - name: Storage3 vip: true name_lower: storage3 ip_subnet: '172.16.33.0/24' allocation_pools: [{'start': '172.16.33.4', 'end': '172.16.33.250'}] ipv6_subnet: 'fd00:fd00:fd00:3003::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:3003::10', 'end': 'fd00:fd00:fd00:3003:ffff:ffff:ffff:fffe'}] - name: InternalApi1 name_lower: internal_api1 vip: true ip_subnet: '172.16.41.0/24' allocation_pools: [{'start': '172.16.41.4', 'end': '172.16.41.250'}] ipv6_subnet: 'fd00:fd00:fd00:2001::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2001::10', 'end': 'fd00:fd00:fd00:2001:ffff:ffff:ffff:fffe'}] - name: InternalApi2 name_lower: internal_api2 vip: true ip_subnet: '172.16.42.0/24' allocation_pools: [{'start': '172.16.42.4', 'end': '172.16.42.250'}] ipv6_subnet: 'fd00:fd00:fd00:2002::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2002::10', 'end': 'fd00:fd00:fd00:2002:ffff:ffff:ffff:fffe'}] - name: InternalApi3 name_lower: internal_api3 vip: true ip_subnet: '172.16.43.0/24' allocation_pools: [{'start': '172.16.43.4', 'end': '172.16.43.250'}] ipv6_subnet: 'fd00:fd00:fd00:2003::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2003::10', 'end': 'fd00:fd00:fd00:2003:ffff:ffff:ffff:fffe'}] - name: Composable1 name_lower: composable1 vip: true ip_subnet: '172.16.44.0/24' allocation_pools: [{'start': '172.16.44.4', 'end': '172.16.44.250'}] ipv6_subnet: 'fd00:fd00:fd00:2004::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2004::10', 'end': 'fd00:fd00:fd00:2004:ffff:ffff:ffff:fffe'}] - name: Composable2 name_lower: composable2 vip: true ipv6: true ip_subnet: '172.16.45.0/24' allocation_pools: [{'start': '172.16.45.4', 'end': '172.16.45.250'}] ipv6_subnet: 'fd00:fd00:fd00:2005::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2005::10', 'end': 'fd00:fd00:fd00:2005:ffff:ffff:ffff:fffe'}] Part of nic config yaml: parameters: InternalApiInterfaceDefaultRoute: # Override this via parameter_defaults description: Default route for the specific network. type: string TenantInterfaceDefaultRoute: # Override this via parameter_defaults description: Default route for the specific network. type: string StorageInterfaceDefaultRoute: # Override this via parameter_defaults description: Default route for the specific network. type: string StorageMgmtInterfaceDefaultRoute: # Override this via parameter_defaults description: Default route for the specific network. type: string InternalApi1NetworkVlanID: default: 21 description: Vlan ID for the internal_api network traffic. type: number InternalApi2NetworkVlanID: default: 22 description: Vlan ID for the internal_api network traffic. type: number InternalApi3NetworkVlanID: default: 23 description: Vlan ID for the internal_api network traffic. type: number Storage1NetworkVlanID: default: 31 description: Vlan ID for the storage network traffic. type: number Storage2NetworkVlanID: default: 32 description: Vlan ID for the storage network traffic. type: number Storage3NetworkVlanID: default: 33 description: Vlan ID for the storage network traffic. type: number StorageMgmt1NetworkVlanID: default: 41 description: Vlan ID for the storage mgmt network traffic. type: number StorageMgmt2NetworkVlanID: default: 42 description: Vlan ID for the storage mgmt network traffic. type: number StorageMgmt3NetworkVlanID: default: 43 description: Vlan ID for the storage mgmt network traffic. type: number Composable1NetworkVlanID: default: 54 description: Vlan ID for the tenant network traffic. type: number Composable2NetworkVlanID: default: 55 description: Vlan ID for the tenant network traffic. type: number InternalApi1IpSubnet: default: '' description: IP address/subnet on the internal API network type: string InternalApi2IpSubnet: default: '' description: IP address/subnet on the internal API network type: string InternalApi3IpSubnet: default: '' description: IP address/subnet on the internal API network type: string Storage1IpSubnet: default: '' description: IP address/subnet on the storage network type: string Storage2IpSubnet: default: '' description: IP address/subnet on the storage network type: string Storage3IpSubnet: default: '' description: IP address/subnet on the storage network type: string StorageMgmt1IpSubnet: default: '' description: IP address/subnet on the storage mgmt network type: string StorageMgmt2IpSubnet: default: '' description: IP address/subnet on the storage mgmt network type: string StorageMgmt3IpSubnet: default: '' description: IP address/subnet on the storage mgmt network type: string Composable1IpSubnet: default: '' description: IP address/subnet on the tenant network type: string Composable2IpSubnet: default: '' description: IP address/subnet on the tenant network type: string ControlPlaneIp: default: '' description: IP address/subnet on the ctlplane network type: string ExternalIpSubnet: default: '' description: IP address/subnet on the external network type: string InternalApiIpSubnet: default: '' description: IP address/subnet on the internal API network type: string StorageIpSubnet: default: '' description: IP address/subnet on the storage network type: string StorageMgmtIpSubnet: default: '' description: IP address/subnet on the storage mgmt network type: string TenantIpSubnet: default: '' description: IP address/subnet on the tenant network type: string ManagementIpSubnet: # Only populated when including environments/network-management.yaml default: '' description: IP address/subnet on the management network type: string ExternalNetworkVlanID: default: 10 description: Vlan ID for the external network traffic. type: number InternalApiNetworkVlanID: default: 20 description: Vlan ID for the internal_api network traffic. type: number StorageNetworkVlanID: default: 30 description: Vlan ID for the storage network traffic. type: number StorageMgmtNetworkVlanID: default: 40 description: Vlan ID for the storage mgmt network traffic. type: number TenantNetworkVlanID: default: 50 description: Vlan ID for the tenant network traffic. type: number ExternalInterfaceDefaultRoute: default: 10.0.0.1 description: default route for the external network type: string ControlPlaneSubnetCidr: # Override this via parameter_defaults default: '24' description: The subnet CIDR of the control plane network. type: string ControlPlaneDefaultRoute: # Override this via parameter_defaults description: The subnet CIDR of the control plane network. type: string DnsServers: # Override this via parameter_defaults default: [] description: A list of DNS servers (2 max for some implementations) that will be added to resolv.conf. type: json EC2MetadataIp: # Override this via parameter_defaults description: The IP address of the EC2 metadata server. type: string Composable1InterfaceRoutes: default: [] description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Composable2InterfaceRoutes: default: [] description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json StorageInterfaceRoutes: default: [] description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json StorageMgmtInterfaceRoutes: default: [] description: 'Routes for the storage_mgmt network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json InternalApiInterfaceRoutes: default: [] description: 'Routes for the internal_api network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json TenantInterfaceRoutes: default: [] description: 'Routes for the tenant network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json ExternalInterfaceRoutes: default: [] description: 'Routes for the external network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json ManagementInterfaceRoutes: default: [] description: 'Routes for the management network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Storage1InterfaceRoutes: default: [] description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json StorageMgmt1InterfaceRoutes: default: [] description: 'Routes for the storage_mgmt network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json InternalApi1InterfaceRoutes: default: [] description: 'Routes for the internal_api network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json External1InterfaceRoutes: default: [] description: 'Routes for the external network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Management1InterfaceRoutes: default: [] description: 'Routes for the management network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Storage2InterfaceRoutes: default: [] description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json StorageMgmt2InterfaceRoutes: default: [] description: 'Routes for the storage_mgmt network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json InternalApi2InterfaceRoutes: default: [] description: 'Routes for the internal_api network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json External2InterfaceRoutes: default: [] description: 'Routes for the external network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Management2InterfaceRoutes: default: [] description: 'Routes for the management network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Storage3InterfaceRoutes: default: [] description: 'Routes for the storage network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json StorageMgmt3InterfaceRoutes: default: [] description: 'Routes for the storage_mgmt network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json InternalApi3InterfaceRoutes: default: [] description: 'Routes for the internal_api network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json External3InterfaceRoutes: default: [] description: 'Routes for the external network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Management3InterfaceRoutes: default: [] description: 'Routes for the management network traffic. JSON route e.g. [{''destination'':''10.0.0.0/16'', ''nexthop'':''10.0.0.1''}] Unless the default is changed, the parameter is automatically resolved from the subnet host_routes attribute. ' type: json Result: The overcloud deployment fails with: "2019-02-20 23:46:15Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller tenant1_ip_address) is incorrect." Tried to remove tenant1 from controller's yaml to see where it takes us. Now there's a new error upon failure: 2019-02-21 03:01:28Z [overcloud.Controller.1]: CREATE_COMPLETE state changed 2019-02-21 03:01:28Z [overcloud.Controller]: UPDATE_COMPLETE Stack UPDATE completed successfully 2019-02-21 03:01:28Z [overcloud.Controller]: CREATE_COMPLETE state changed 2019-02-21 03:01:30Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller tenant1_ip_address) is incorrect. 2019-02-21 03:01:31Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller storage_mgmt3_ip_address) is incorrect. 2019-02-21 03:01:31Z [overcloud]: CREATE_FAILED The Referenced Attribute (Controller storage_mgmt2_ip_address) is incorrect. Stack overcloud/1842300f-d20a-4af8-bbb5-3ba7791fafa5 CREATE_FAILED Heat Stack create failed. Heat Stack create failed. This happens pretty early - before the os-net-config (the OC nodes booted, but the proper addresses wasn't set on nodes).
Applying the changes in this patch https://review.openstack.org/#/c/638440/ fixed the issue and the deployment succeeded.
As this is a regression putting blocker flag back on. We'd like to get this into z1 if possible.
Verified. Environment: openstack-tripleo-heat-templates-9.2.1-0.20190119154863.el7ost.noarch The reported issue doesn't reproduce.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0446