Bug 1628358 - Overcloud deployment with predictable IPs and custom hostnames fails with The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
Summary: Overcloud deployment with predictable IPs and custom hostnames fails with The...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: beta
: 14.0 (Rocky)
Assignee: Harald Jensås
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-09-12 19:58 UTC by Marius Cornea
Modified: 2019-01-11 11:53 UTC (History)
10 users (show)

Fixed In Version: openstack-tripleo-heat-templates-9.0.0-0.20180919080945.0rc1.0rc1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-11 11:53:06 UTC
Target Upstream Version:
Embargoed:
hjensas: needinfo-


Attachments (Terms of Use)
heat-logs (1.86 MB, application/x-gzip)
2018-09-12 19:59 UTC, Marius Cornea
no flags Details
virt.tar.gz (11.05 KB, application/x-gzip)
2018-09-12 20:01 UTC, Marius Cornea
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1792968 0 None None None 2018-09-17 15:53:30 UTC
OpenStack gerrit 603172 0 None None None 2018-09-17 15:53:05 UTC
OpenStack gerrit 604354 0 None None None 2018-09-26 12:43:27 UTC
Red Hat Product Errata RHEA-2019:0045 0 None None None 2019-01-11 11:53:16 UTC

Description Marius Cornea 2018-09-12 19:58:22 UTC
Description of problem:
Overcloud deployment with predictable IPs and custom hostnames fails with The Referenced Attribute (StorageMgmtPort host_routes) is incorrect:

2018-09-12 18:06:30Z [qe-Cloud-0.Controller.2]: CREATE_FAILED  The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
2018-09-12 18:06:31Z [qe-Cloud-0.Controller.1.InternalApiPort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:31Z [qe-Cloud-0.Compute.0.InternalApiPort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:31Z [qe-Cloud-0.Compute.0.StoragePort]: CREATE_IN_PROGRESS  state changed
2018-09-12 18:06:32Z [qe-Cloud-0.Controller.1.StoragePort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:34Z [qe-Cloud-0.Compute.1.ManagementPort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:34Z [qe-Cloud-0.Compute.0.PreNetworkConfig]: CREATE_IN_PROGRESS  state changed
2018-09-12 18:06:34Z [qe-Cloud-0.Compute.0.StorageMgmtPort]: CREATE_IN_PROGRESS  state changed
2018-09-12 18:06:34Z [qe-Cloud-0.Compute.1.TenantPort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:34Z [qe-Cloud-0.Controller.2]: CREATE_FAILED  resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
2018-09-12 18:06:34Z [qe-Cloud-0.Compute.0.TenantPort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:34Z [qe-Cloud-0.Controller]: UPDATE_FAILED  Resource CREATE failed: resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
2018-09-12 18:06:35Z [qe-Cloud-0.Compute.0.ExternalPort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:35Z [qe-Cloud-0.Controller]: CREATE_FAILED  resources.Controller: Resource CREATE failed: resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
2018-09-12 18:06:35Z [qe-Cloud-0]: CREATE_FAILED  Resource CREATE failed: resources.Controller: Resource CREATE failed: resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
2018-09-12 18:06:36Z [qe-Cloud-0.Compute.0.ManagementPort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:36Z [qe-Cloud-0.CephStorage.2.StorageMgmtPort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:37Z [qe-Cloud-0.CephStorage.2.TenantPort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:37Z [qe-Cloud-0.Compute.0.StoragePort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:38Z [qe-Cloud-0.Compute.0.PreNetworkConfig]: CREATE_COMPLETE  state changed
2018-09-12 18:06:38Z [qe-Cloud-0.Controller.1.ExternalPort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:38Z [qe-Cloud-0.Compute.1.NetIpMap]: CREATE_IN_PROGRESS  state changed
2018-09-12 18:06:38Z [qe-Cloud-0.Compute.1]: CREATE_FAILED  The Referenced Attribute (TenantPort host_routes) is incorrect.
2018-09-12 18:06:38Z [qe-Cloud-0.Controller.1.TenantPort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:39Z [qe-Cloud-0.Controller.1.ManagementPort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:39Z [qe-Cloud-0.Compute.1]: CREATE_FAILED  resources[1]: The Referenced Attribute (TenantPort host_routes) is incorrect.
2018-09-12 18:06:39Z [qe-Cloud-0.CephStorage.2.StoragePort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:39Z [qe-Cloud-0.Compute]: UPDATE_FAILED  Resource CREATE failed: resources[1]: The Referenced Attribute (TenantPort host_routes) is incorrect.
2018-09-12 18:06:39Z [qe-Cloud-0.CephStorage.2]: CREATE_FAILED  The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
2018-09-12 18:06Heat Stack create failed.
Heat Stack create failed.
:39Z [qe-Cloud-0.Compute]: CREATE_FAILED  resources.Compute: Resource CREATE failed: resources[1]: The Referenced Attribute (TenantPort host_routes) is incorrect.
2018-09-12 18:06:39Z [qe-Cloud-0.CephStorage.2.NetIpMap]: CREATE_IN_PROGRESS  state changed
2018-09-12 18:06:39Z [qe-Cloud-0.CephStorage.2]: CREATE_FAILED  resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
2018-09-12 18:06:40Z [qe-Cloud-0]: CREATE_FAILED  Resource CREATE failed: resources.Compute: Resource CREATE failed: resources[1]: The Referenced Attribute (TenantPort host_routes) is incorrect.
2018-09-12 18:06:40Z [qe-Cloud-0.Controller.1]: CREATE_FAILED  The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
2018-09-12 18:06:40Z [qe-Cloud-0.CephStorage]: UPDATE_FAILED  Resource CREATE failed: resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
2018-09-12 18:06:40Z [qe-Cloud-0.Controller.1.NetIpMap]: CREATE_IN_PROGRESS  state changed
2018-09-12 18:06:40Z [qe-Cloud-0.Controller.2.NetIpMap]: CREATE_COMPLETE  state changed
2018-09-12 18:06:40Z [qe-Cloud-0.CephStorage]: CREATE_FAILED  resources.CephStorage: Resource CREATE failed: resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
2018-09-12 18:06:42Z [qe-Cloud-0.Controller.1]: CREATE_FAILED  resources[1]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
2018-09-12 18:06:42Z [qe-Cloud-0.Controller]: UPDATE_FAILED  Resource CREATE failed: resources[1]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
2018-09-12 18:06:42Z [qe-Cloud-0]: CREATE_FAILED  Resource CREATE failed: resources.CephStorage: Resource CREATE failed: resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
2018-09-12 18:06:42Z [qe-Cloud-0.Compute.0.StorageMgmtPort]: CREATE_COMPLETE  state changed
2018-09-12 18:06:42Z [qe-Cloud-0.Compute.0]: CREATE_FAILED  The Referenced Attribute (TenantPort host_routes) is incorrect.
2018-09-12 18:06:43Z [qe-Cloud-0.Compute.0.NetIpMap]: CREATE_IN_PROGRESS  state changed
2018-09-12 18:06:43Z [qe-Cloud-0.Compute.0]: CREATE_FAILED  resources[0]: The Referenced Attribute (TenantPort host_routes) is incorrect.
2018-09-12 18:06:43Z [qe-Cloud-0.Compute]: UPDATE_FAILED  Resource CREATE failed: resources[0]: The Referenced Attribute (TenantPort host_routes) is incorrect.
2018-09-12 18:06:43Z [qe-Cloud-0.CephStorage.2.NetIpMap]: CREATE_COMPLETE  state changed
2018-09-12 18:06:43Z [qe-Cloud-0.Controller.1.NetIpMap]: CREATE_COMPLETE  state changed
2018-09-12 18:06:43Z [qe-Cloud-0.Compute.1.NetIpMap]: CREATE_COMPLETE  state changed
2018-09-12 18:06:46Z [qe-Cloud-0.Compute.0.NetIpMap]: CREATE_COMPLETE  state changed

 Stack qe-Cloud-0/7f550ecb-be49-4455-9511-1c4c1b14ecb5 CREATE_FAILED 

qe-Cloud-0.Controller.1:
  resource_type: OS::TripleO::Controller
  physical_resource_id: 039925b5-fcbe-469c-bae0-90af7d264202
  status: CREATE_FAILED
  status_reason: |
    resources[1]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
qe-Cloud-0.Controller.2:
  resource_type: OS::TripleO::Controller
  physical_resource_id: 20e9fb0d-c2f7-41cd-aed8-9c8abc0024a8
  status: CREATE_FAILED
  status_reason: |
    resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.
qe-Cloud-0.Compute.1:
  resource_type: OS::TripleO::Compute
  physical_resource_id: f1dde996-134f-4ea0-802a-d6a5bb66449e
  status: CREATE_FAILED
  status_reason: |
    resources[1]: The Referenced Attribute (TenantPort host_routes) is incorrect.
qe-Cloud-0.Compute.0:
  resource_type: OS::TripleO::Compute
  physical_resource_id: 192e40bc-338b-4142-a33d-487b713a1e27
  status: CREATE_FAILED
  status_reason: |
    resources[0]: The Referenced Attribute (TenantPort host_routes) is incorrect.
qe-Cloud-0.CephStorage.2:
  resource_type: OS::TripleO::CephStorage
  physical_resource_id: 9b4374e3-445f-433d-92f5-9975a0a2699f
  status: CREATE_FAILED
  status_reason: |
    resources[2]: The Referenced Attribute (StorageMgmtPort host_routes) is incorrect.


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-9.0.0-0.20180831204457.17bb71e.0rc1.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack qe-Cloud-0 \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/internal.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/predictable_ips_rr.yml \
-e /home/stack/virt/network/predictable-ips.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/enable-tls.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
-e /home/stack/virt/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/custom_hostnames.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-images.yaml \


Actual results:
Fails with aforementioned error.

Expected results:
No failure.

Additional info:
Attaching heat logs and files used for deploy command.

Comment 1 Marius Cornea 2018-09-12 19:59:48 UTC
Created attachment 1482837 [details]
heat-logs

Comment 2 Marius Cornea 2018-09-12 20:01:28 UTC
Created attachment 1482838 [details]
virt.tar.gz

templates

Comment 3 Harald Jensås 2018-09-17 15:31:29 UTC
This is due to this change:
https://review.openstack.org/#/c/580596/6/puppet/role.role.j2.yaml

We use get_attr there to populate fetch the routes from the neutron subnet/port info. We need the fake port created by THT/network/ports/port_from_pool.j2 to output the host_routes just like the THT/network/ports/port.j2 does.

I will create an upstream bug and propose a fix.

Comment 4 Harald Jensås 2018-09-17 15:43:01 UTC
Another way to fix this is to drop the resource overrides for TripleO Port resources. See: https://review.openstack.org/580747.

Comment 11 Alexander Chuzhoy 2018-10-16 18:45:58 UTC
Verified:

Environment:
openstack-tripleo-heat-templates-9.0.0-0.20180919080945.0rc1.0rc1.el7ost.noarch

The reported issue doesn't reproduce and the oc deployment completes successfully.

Comment 14 errata-xmlrpc 2019-01-11 11:53:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045


Note You need to log in before you can comment on or make changes to this bug.