Description of problem: Overcloud deployment with predictable IPs and custom hostnames fails with The Referenced Attribute (StorageMgmtPort host_routes) is incorrect: 2018-09-12 18:06:30Z [qe-Cloud-0.Controller.2]: CREATE_FAILED The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. 2018-09-12 18:06:31Z [qe-Cloud-0.Controller.1.InternalApiPort]: CREATE_COMPLETE state changed 2018-09-12 18:06:31Z [qe-Cloud-0.Compute.0.InternalApiPort]: CREATE_COMPLETE state changed 2018-09-12 18:06:31Z [qe-Cloud-0.Compute.0.StoragePort]: CREATE_IN_PROGRESS state changed 2018-09-12 18:06:32Z [qe-Cloud-0.Controller.1.StoragePort]: CREATE_COMPLETE state changed 2018-09-12 18:06:34Z [qe-Cloud-0.Compute.1.ManagementPort]: CREATE_COMPLETE state changed 2018-09-12 18:06:34Z [qe-Cloud-0.Compute.0.PreNetworkConfig]: CREATE_IN_PROGRESS state changed 2018-09-12 18:06:34Z [qe-Cloud-0.Compute.0.StorageMgmtPort]: CREATE_IN_PROGRESS state changed 2018-09-12 18:06:34Z [qe-Cloud-0.Compute.1.TenantPort]: CREATE_COMPLETE state changed 2018-09-12 18:06:34Z [qe-Cloud-0.Controller.2]: CREATE_FAILED resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. 2018-09-12 18:06:34Z [qe-Cloud-0.Compute.0.TenantPort]: CREATE_COMPLETE state changed 2018-09-12 18:06:34Z [qe-Cloud-0.Controller]: UPDATE_FAILED Resource CREATE failed: resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. 2018-09-12 18:06:35Z [qe-Cloud-0.Compute.0.ExternalPort]: CREATE_COMPLETE state changed 2018-09-12 18:06:35Z [qe-Cloud-0.Controller]: CREATE_FAILED resources.Controller: Resource CREATE failed: resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. 2018-09-12 18:06:35Z [qe-Cloud-0]: CREATE_FAILED Resource CREATE failed: resources.Controller: Resource CREATE failed: resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. 2018-09-12 18:06:36Z [qe-Cloud-0.Compute.0.ManagementPort]: CREATE_COMPLETE state changed 2018-09-12 18:06:36Z [qe-Cloud-0.CephStorage.2.StorageMgmtPort]: CREATE_COMPLETE state changed 2018-09-12 18:06:37Z [qe-Cloud-0.CephStorage.2.TenantPort]: CREATE_COMPLETE state changed 2018-09-12 18:06:37Z [qe-Cloud-0.Compute.0.StoragePort]: CREATE_COMPLETE state changed 2018-09-12 18:06:38Z [qe-Cloud-0.Compute.0.PreNetworkConfig]: CREATE_COMPLETE state changed 2018-09-12 18:06:38Z [qe-Cloud-0.Controller.1.ExternalPort]: CREATE_COMPLETE state changed 2018-09-12 18:06:38Z [qe-Cloud-0.Compute.1.NetIpMap]: CREATE_IN_PROGRESS state changed 2018-09-12 18:06:38Z [qe-Cloud-0.Compute.1]: CREATE_FAILED The Referenced Attribute (TenantPort host_routes) is incorrect. 2018-09-12 18:06:38Z [qe-Cloud-0.Controller.1.TenantPort]: CREATE_COMPLETE state changed 2018-09-12 18:06:39Z [qe-Cloud-0.Controller.1.ManagementPort]: CREATE_COMPLETE state changed 2018-09-12 18:06:39Z [qe-Cloud-0.Compute.1]: CREATE_FAILED resources[1]: The Referenced Attribute (TenantPort host_routes) is incorrect. 2018-09-12 18:06:39Z [qe-Cloud-0.CephStorage.2.StoragePort]: CREATE_COMPLETE state changed 2018-09-12 18:06:39Z [qe-Cloud-0.Compute]: UPDATE_FAILED Resource CREATE failed: resources[1]: The Referenced Attribute (TenantPort host_routes) is incorrect. 2018-09-12 18:06:39Z [qe-Cloud-0.CephStorage.2]: CREATE_FAILED The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. 2018-09-12 18:06Heat Stack create failed. Heat Stack create failed. :39Z [qe-Cloud-0.Compute]: CREATE_FAILED resources.Compute: Resource CREATE failed: resources[1]: The Referenced Attribute (TenantPort host_routes) is incorrect. 2018-09-12 18:06:39Z [qe-Cloud-0.CephStorage.2.NetIpMap]: CREATE_IN_PROGRESS state changed 2018-09-12 18:06:39Z [qe-Cloud-0.CephStorage.2]: CREATE_FAILED resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. 2018-09-12 18:06:40Z [qe-Cloud-0]: CREATE_FAILED Resource CREATE failed: resources.Compute: Resource CREATE failed: resources[1]: The Referenced Attribute (TenantPort host_routes) is incorrect. 2018-09-12 18:06:40Z [qe-Cloud-0.Controller.1]: CREATE_FAILED The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. 2018-09-12 18:06:40Z [qe-Cloud-0.CephStorage]: UPDATE_FAILED Resource CREATE failed: resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. 2018-09-12 18:06:40Z [qe-Cloud-0.Controller.1.NetIpMap]: CREATE_IN_PROGRESS state changed 2018-09-12 18:06:40Z [qe-Cloud-0.Controller.2.NetIpMap]: CREATE_COMPLETE state changed 2018-09-12 18:06:40Z [qe-Cloud-0.CephStorage]: CREATE_FAILED resources.CephStorage: Resource CREATE failed: resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. 2018-09-12 18:06:42Z [qe-Cloud-0.Controller.1]: CREATE_FAILED resources[1]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. 2018-09-12 18:06:42Z [qe-Cloud-0.Controller]: UPDATE_FAILED Resource CREATE failed: resources[1]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. 2018-09-12 18:06:42Z [qe-Cloud-0]: CREATE_FAILED Resource CREATE failed: resources.CephStorage: Resource CREATE failed: resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. 2018-09-12 18:06:42Z [qe-Cloud-0.Compute.0.StorageMgmtPort]: CREATE_COMPLETE state changed 2018-09-12 18:06:42Z [qe-Cloud-0.Compute.0]: CREATE_FAILED The Referenced Attribute (TenantPort host_routes) is incorrect. 2018-09-12 18:06:43Z [qe-Cloud-0.Compute.0.NetIpMap]: CREATE_IN_PROGRESS state changed 2018-09-12 18:06:43Z [qe-Cloud-0.Compute.0]: CREATE_FAILED resources[0]: The Referenced Attribute (TenantPort host_routes) is incorrect. 2018-09-12 18:06:43Z [qe-Cloud-0.Compute]: UPDATE_FAILED Resource CREATE failed: resources[0]: The Referenced Attribute (TenantPort host_routes) is incorrect. 2018-09-12 18:06:43Z [qe-Cloud-0.CephStorage.2.NetIpMap]: CREATE_COMPLETE state changed 2018-09-12 18:06:43Z [qe-Cloud-0.Controller.1.NetIpMap]: CREATE_COMPLETE state changed 2018-09-12 18:06:43Z [qe-Cloud-0.Compute.1.NetIpMap]: CREATE_COMPLETE state changed 2018-09-12 18:06:46Z [qe-Cloud-0.Compute.0.NetIpMap]: CREATE_COMPLETE state changed Stack qe-Cloud-0/7f550ecb-be49-4455-9511-1c4c1b14ecb5 CREATE_FAILED qe-Cloud-0.Controller.1: resource_type: OS::TripleO::Controller physical_resource_id: 039925b5-fcbe-469c-bae0-90af7d264202 status: CREATE_FAILED status_reason: | resources[1]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. qe-Cloud-0.Controller.2: resource_type: OS::TripleO::Controller physical_resource_id: 20e9fb0d-c2f7-41cd-aed8-9c8abc0024a8 status: CREATE_FAILED status_reason: | resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. qe-Cloud-0.Compute.1: resource_type: OS::TripleO::Compute physical_resource_id: f1dde996-134f-4ea0-802a-d6a5bb66449e status: CREATE_FAILED status_reason: | resources[1]: The Referenced Attribute (TenantPort host_routes) is incorrect. qe-Cloud-0.Compute.0: resource_type: OS::TripleO::Compute physical_resource_id: 192e40bc-338b-4142-a33d-487b713a1e27 status: CREATE_FAILED status_reason: | resources[0]: The Referenced Attribute (TenantPort host_routes) is incorrect. qe-Cloud-0.CephStorage.2: resource_type: OS::TripleO::CephStorage physical_resource_id: 9b4374e3-445f-433d-92f5-9975a0a2699f status: CREATE_FAILED status_reason: | resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect. Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-9.0.0-0.20180831204457.17bb71e.0rc1.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. openstack overcloud deploy \ --timeout 100 \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack qe-Cloud-0 \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /home/stack/virt/internal.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/predictable_ips_rr.yml \ -e /home/stack/virt/network/predictable-ips.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/virt/enable-tls.yaml \ -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/virt/public_vip.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \ -e /home/stack/virt/hostnames.yml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/custom_hostnames.yaml \ -e /home/stack/virt/nodes_data.yaml \ -e /home/stack/virt/docker-images.yaml \ Actual results: Fails with aforementioned error. Expected results: No failure. Additional info: Attaching heat logs and files used for deploy command.
Created attachment 1482837 [details] heat-logs
Created attachment 1482838 [details] virt.tar.gz templates
This is due to this change: https://review.openstack.org/#/c/580596/6/puppet/role.role.j2.yaml We use get_attr there to populate fetch the routes from the neutron subnet/port info. We need the fake port created by THT/network/ports/port_from_pool.j2 to output the host_routes just like the THT/network/ports/port.j2 does. I will create an upstream bug and propose a fix.
Another way to fix this is to drop the resource overrides for TripleO Port resources. See: https://review.openstack.org/580747.
Verified: Environment: openstack-tripleo-heat-templates-9.0.0-0.20180919080945.0rc1.0rc1.el7ost.noarch The reported issue doesn't reproduce and the oc deployment completes successfully.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045