Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1731260 - Doc: ComputeExtraConfig to avoid errors like: Failing to launch instance on an OC deployed with spine&leaf network: F ailed to bind port 11072725-8eea-4e78-af49-e8079df5fdf7 on host overcloud-novacompute2-0.localdomain for vnic_type normal using segments
Summary: Doc: ComputeExtraConfig to avoid errors like: Failing to launch instance on a...
Keywords:
Status: CLOSED EOL
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: documentation
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: RHOS Documentation Team
QA Contact: RHOS Documentation Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-18 19:32 UTC by Alexander Chuzhoy
Modified: 2021-07-07 09:20 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-07 09:20:56 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Alexander Chuzhoy 2019-07-18 19:32:51 UTC
Environment:
puppet-neutron-14.4.1-0.20190531220405.ff3610d.el8ost.noarch
python3-neutron-lib-1.25.0-0.20190521130309.fc2a810.el8ost.noarch
python3-neutron-14.0.3-0.20190704180411.9f4e596.el8ost.noarch
openstack-neutron-common-14.0.3-0.20190704180411.9f4e596.el8ost.noarch
python3-neutron-dynamic-routing-14.0.1-0.20190426180400.f313f0e.1.el8ost.noarch
openstack-neutron-lbaas-14.0.1-0.20190614170521.30bdd86.el8ost.noarch
openstack-neutron-ml2-14.0.3-0.20190704180411.9f4e596.el8ost.noarch
python3-neutronclient-6.12.0-0.20190312100012.680b417.el8ost.noarch
openstack-neutron-14.0.3-0.20190704180411.9f4e596.el8ost.noarch
python3-neutron-lbaas-14.0.1-0.20190614170521.30bdd86.el8ost.noarch


Steps to reproduce:
Deploy OC with spine&leaf




Attempt to launch an instance in OC.


Result:
The instance is unable to launch - gets status error:
| fault                               | {'code': 500, 'created': '2019-07-18T19:24:39Z', 'message': 'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 512e3d6d-3493-4905-ab9b-9d30e2223822. Last exception: Binding failed for port 8ced21dc-bf03-46ba-b998-0f06d6e8c45c, please check neutron logs for more information.', 'details': '  File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 619, in build_instances\n    filter_properties, instances[0].uuid)\n  File "/usr/lib/python3.6/site-packages/nova/scheduler/utils.py", line 757, in populate_retry\n    raise exception.MaxRetriesExceeded(reason=msg)\n'} |



Looking in neutron logs:

2019-07-18 19:05:18.102 22 ERROR neutron.plugins.ml2.managers [req-39ba616b-f606-4ec4-b84d-422163bc3875 143dfcc455d54d98a2f3ba1439499ad0 d86dbfd69cd547cd97443154f63538db - default default] F
ailed to bind port 11072725-8eea-4e78-af49-e8079df5fdf7 on host overcloud-novacompute2-0.localdomain for vnic_type normal using segments [{'id': 'f3253a93-fb66-4d8f-92cf-13202b575f19', 'netw
ork_type': 'geneve', 'physical_network': None, 'segmentation_id': 10, 'network_id': '7544d1b0-e8ba-4eda-a8fe-518074774415'}]
2019-07-18 19:05:18.103 22 INFO neutron.plugins.ml2.plugin [req-39ba616b-f606-4ec4-b84d-422163bc3875 143dfcc455d54d98a2f3ba1439499ad0 d86dbfd69cd547cd97443154f63538db - default default] Atte
mpt 5 to bind port 11072725-8eea-4e78-af49-e8079df5fdf7
2019-07-18 19:05:18.120 22 ERROR neutron.plugins.ml2.managers [req-39ba616b-f606-4ec4-b84d-422163bc3875 143dfcc455d54d98a2f3ba1439499ad0 d86dbfd69cd547cd97443154f63538db - default default] F
ailed to bind port 11072725-8eea-4e78-af49-e8079df5fdf7 on host overcloud-novacompute2-0.localdomain for vnic_type normal using segments [{'id': 'f3253a93-fb66-4d8f-92cf-13202b575f19', 'netw
ork_type': 'geneve', 'physical_network': None, 'segmentation_id': 10, 'network_id': '7544d1b0-e8ba-4eda-a8fe-518074774415'}]
2019-07-18 19:05:18.120 22 INFO neutron.plugins.ml2.plugin [req-39ba616b-f606-4ec4-b84d-422163bc3875 143dfcc455d54d98a2f3ba1439499ad0 d86dbfd69cd547cd97443154f63538db - default default] Atte
mpt 6 to bind port 11072725-8eea-4e78-af49-e8079df5fdf7
2019-07-18 19:05:18.139 22 ERROR neutron.plugins.ml2.managers [req-39ba616b-f606-4ec4-b84d-422163bc3875 143dfcc455d54d98a2f3ba1439499ad0 d86dbfd69cd547cd97443154f63538db - default default] F
ailed to bind port 11072725-8eea-4e78-af49-e8079df5fdf7 on host overcloud-novacompute2-0.localdomain for vnic_type normal using segments [{'id': 'f3253a93-fb66-4d8f-92cf-13202b575f19', 'netw
ork_type': 'geneve', 'physical_network': None, 'segmentation_id': 10, 'network_id': '7544d1b0-e8ba-4eda-a8fe-518074774415'}]
2019-07-18 19:05:18.139 22 INFO neutron.plugins.ml2.plugin [req-39ba616b-f606-4ec4-b84d-422163bc3875 143dfcc455d54d98a2f3ba1439499ad0 d86dbfd69cd547cd97443154f63538db - default default] Atte
mpt 7 to bind port 11072725-8eea-4e78-af49-e8079df5fdf7
2019-07-18 19:05:18.156 22 ERROR neutron.plugins.ml2.managers [req-39ba616b-f606-4ec4-b84d-422163bc3875 143dfcc455d54d98a2f3ba1439499ad0 d86dbfd69cd547cd97443154f63538db - default default] F
ailed to bind port 11072725-8eea-4e78-af49-e8079df5fdf7 on host overcloud-novacompute2-0.localdomain for vnic_type normal using segments [{'id': 'f3253a93-fb66-4d8f-92cf-13202b575f19', 'netw
ork_type': 'geneve', 'physical_network': None, 'segmentation_id': 10, 'network_id': '7544d1b0-e8ba-4eda-a8fe-518074774415'}]
2019-07-18 19:05:18.156 22 INFO neutron.plugins.ml2.plugin [req-39ba616b-f606-4ec4-b84d-422163bc3875 143dfcc455d54d98a2f3ba1439499ad0 d86dbfd69cd547cd97443154f63538db - default default] Atte
mpt 8 to bind port 11072725-8eea-4e78-af49-e8079df5fdf7
2019-07-18 19:05:18.173 22 ERROR neutron.plugins.ml2.managers [req-39ba616b-f606-4ec4-b84d-422163bc3875 143dfcc455d54d98a2f3ba1439499ad0 d86dbfd69cd547cd97443154f63538db - default default] F
ailed to bind port 11072725-8eea-4e78-af49-e8079df5fdf7 on host overcloud-novacompute2-0.localdomain for vnic_type normal using segments [{'id': 'f3253a93-fb66-4d8f-92cf-13202b575f19', 'netw
ork_type': 'geneve', 'physical_network': None, 'segmentation_id': 10, 'network_id': '7544d1b0-e8ba-4eda-a8fe-518074774415'}]
2019-07-18 19:05:18.174 22 INFO neutron.plugins.ml2.plugin [req-39ba616b-f606-4ec4-b84d-422163bc3875 143dfcc455d54d98a2f3ba1439499ad0 d86dbfd69cd547cd97443154f63538db - default default] Attempt 9 to bind port 11072725-8eea-4e78-af49-e8079df5fdf7
2019-07-18 19:05:18.191 22 ERROR neutron.plugins.ml2.managers [req-39ba616b-f606-4ec4-b84d-422163bc3875 143dfcc455d54d98a2f3ba1439499ad0 d86dbfd69cd547cd97443154f63538db - default default] Failed to bind port 11072725-8eea-4e78-af49-e8079df5fdf7 on host overcloud-novacompute2-0.localdomain for vnic_type normal using segments [{'id': 'f3253a93-fb66-4d8f-92cf-13202b575f19', 'network_type': 'geneve', 'physical_network': None, 'segmentation_id': 10, 'network_id': '7544d1b0-e8ba-4eda-a8fe-518074774415'}]
2019-07-18 19:05:18.192 22 INFO neutron.plugins.ml2.plugin [req-39ba616b-f606-4ec4-b84d-422163bc3875 143dfcc455d54d98a2f3ba1439499ad0 d86dbfd69cd547cd97443154f63538db - default default] Attempt 10 to bind port 11072725-8eea-4e78-af49-e8079df5fdf7
2019-07-18 19:05:18.210 22 ERROR neutron.plugins.ml2.managers [req-39ba616b-f606-4ec4-b84d-422163bc3875 143dfcc455d54d98a2f3ba1439499ad0 d86dbfd69cd547cd97443154f63538db - default default] Failed to bind port 11072725-8eea-4e78-af49-e8079df5fdf7 on host overcloud-novacompute2-0.localdomain for vnic_type normal using segments [{'id': 'f3253a93-fb66-4d8f-92cf-13202b575f19', 'network_type': 'geneve', 'physical_network': None, 'segmentation_id': 10, 'network_id': '7544d1b0-e8ba-4eda-a8fe-518074774415'}]

Comment 2 Harald Jensås 2019-07-18 22:43:04 UTC
I noticed this in the /etc/hosts file:


 overcloud-novacompute1-0.localdomain overcloud-novacompute1-0
172.117.2.75 overcloud-novacompute1-0.tenant1.localdomain overcloud-
novacompute1-0.tenant1

 overcloud-novacompute1-1.localdomain overcloud-novacompute1-1
172.117.2.46 overcloud-novacompute1-1.tenant1.localdomain overcloud-
novacompute1-1.tenant1


See how the .localdomain entries for the nodes on the leaf's don't have
an IP address entry?

To get those added I belive you need to add this to your environment:

parameters_default:
  Compute1HostnameResolveNetwork: internal_api1
  Compute2HostnameResolveNetwork: internal_api2
  Compute3HostnameResolveNetwork: internal_api3
  CephStorage1HostnameResolveNetwork: storage1
  CephStorage2HostnameResolveNetwork: storage2
  CephStorage3HostnameResolveNetwork: storage3

There are a few config files that use those names:
[root@overcloud-novacompute2-0 ~]# grep -R overcloud-novacompute2-
0.localdomain /var/lib/config-data/puppet-generated/
/var/lib/config-data/puppet-
generated/ceilometer/etc/ceilometer/ceilometer.conf:host=overcloud-
novacompute2-0.localdomain
/var/lib/config-data/puppet-
generated/neutron/etc/neutron/neutron.conf:host=overcloud-novacompute2-
0.localdomain
/var/lib/config-data/puppet-
generated/nova_libvirt/etc/nova/nova.conf:host=overcloud-novacompute2-
0.localdomain


I can't tell if this is the reason for the port bind failure, but let's
try fixing it?


There is also an error in the ovn-metadata-agent.log


overcloud-novacompute2-0 - /var/log/containers/neutron/ovn-metadata-agent.log

2019-07-18 22:35:46.906 339796 CRITICAL neutron [-] Unhandled error: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=ee6e89cc-9683-4b7c-a7f7-120df9bb6930
2019-07-18 22:35:46.906 339796 ERROR neutron Traceback (most recent call last):
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 104, in transaction
2019-07-18 22:35:46.906 339796 ERROR neutron     yield self._nested_txns_map[cur_thread_id]
2019-07-18 22:35:46.906 339796 ERROR neutron KeyError: 139943364705320
2019-07-18 22:35:46.906 339796 ERROR neutron 
2019-07-18 22:35:46.906 339796 ERROR neutron During handling of the above exception, another exception occurred:
2019-07-18 22:35:46.906 339796 ERROR neutron 
2019-07-18 22:35:46.906 339796 ERROR neutron Traceback (most recent call last):
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/bin/networking-ovn-metadata-agent", line 10, in <module>
2019-07-18 22:35:46.906 339796 ERROR neutron     sys.exit(main())
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib/python3.6/site-packages/networking_ovn/cmd/eventlet/agents/metadata.py", line 17, in main
2019-07-18 22:35:46.906 339796 ERROR neutron     metadata_agent.main()
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib/python3.6/site-packages/networking_ovn/agent/metadata_agent.py", line 38, in main
2019-07-18 22:35:46.906 339796 ERROR neutron     agt.start()
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib/python3.6/site-packages/networking_ovn/agent/metadata/agent.py", line 189, in start
2019-07-18 22:35:46.906 339796 ERROR neutron     self._register_metadata_agent()
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib/python3.6/site-packages/networking_ovn/agent/metadata/agent.py", line 199, in _register_metadata_agent
2019-07-18 22:35:46.906 339796 ERROR neutron     ext_ids).execute(check_error=True)
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute
2019-07-18 22:35:46.906 339796 ERROR neutron     txn.add(self)
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2019-07-18 22:35:46.906 339796 ERROR neutron     next(self.gen)
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 112, in transaction
2019-07-18 22:35:46.906 339796 ERROR neutron     del self._nested_txns_map[cur_thread_id]
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
2019-07-18 22:35:46.906 339796 ERROR neutron     self.result = self.commit()
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2019-07-18 22:35:46.906 339796 ERROR neutron     raise result.ex
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
2019-07-18 22:35:46.906 339796 ERROR neutron     txn.results.put(txn.do_commit())
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit
2019-07-18 22:35:46.906 339796 ERROR neutron     command.run_idl(txn)
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 144, in run_idl
2019-07-18 22:35:46.906 339796 ERROR neutron     record = self.api.lookup(self.table, self.record)
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 103, in lookup
2019-07-18 22:35:46.906 339796 ERROR neutron     return self._lookup(table, record)
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 143, in _lookup
2019-07-18 22:35:46.906 339796 ERROR neutron     row = idlutils.row_by_value(self, rl.table, rl.column, record)
2019-07-18 22:35:46.906 339796 ERROR neutron   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 65, in row_by_value
2019-07-18 22:35:46.906 339796 ERROR neutron     raise RowNotFound(table=table, col=column, match=match)
2019-07-18 22:35:46.906 339796 ERROR neutron ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=ee6e89cc-9683-4b7c-a7f7-120df9bb6930
2019-07-18 22:35:46.906 339796 ERROR neutron 
2019-07-18 22:35:47.198 339809 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting
2019-07-18 22:35:47.199 339809 INFO eventlet.wsgi.server [-] (339809) wsgi exited, is_accepting=True

Comment 3 Harald Jensås 2019-07-19 10:58:00 UTC
(In reply to Harald Jensås from comment #2)

This example is incorrect:

> parameters_default:
>   Compute1HostnameResolveNetwork: internal_api1
>   Compute2HostnameResolveNetwork: internal_api2
>   Compute3HostnameResolveNetwork: internal_api3
>   CephStorage1HostnameResolveNetwork: storage1
>   CephStorage2HostnameResolveNetwork: storage2
>   CephStorage3HostnameResolveNetwork: storage3
> 

These properties are in the ServiceNetMap so it must be:

parameter_defaults:
  ServiceNetMap:
    Compute1HostnameResolveNetwork: internal_api1
    Compute2HostnameResolveNetwork: internal_api2
    Compute3HostnameResolveNetwork: internal_api3
    CephStorage1HostnameResolveNetwork: storage1
    CephStorage2HostnameResolveNetwork: storage2
    CephStorage3HostnameResolveNetwork: storage3


After re-running the deployment with these added to the ServiceNetMap the /etc/hosts file looks good:

(undercloud) [stack@undercloud-0 ~]$ ssh heat-admin@192.168.24.24 'cat /etc/hosts'
Warning: Permanently added '192.168.24.24' (ECDSA) to the list of known hosts.
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6


# HEAT_HOSTS_START - Do not edit manually within this section!
192.168.24.15  overcloud.ctlplane.localdomain
10.0.0.101  overcloud.localdomain
172.120.1.51  overcloud.internalapi.localdomain
172.120.3.44  overcloud.storage.localdomain
172.120.4.102  overcloud.storagemgmt.localdomain
172.120.1.170 overcloud-controller-0.localdomain overcloud-controller-0
10.0.0.117 overcloud-controller-0.external.localdomain overcloud-controller-0.external
172.120.1.170 overcloud-controller-0.internalapi.localdomain overcloud-controller-0.internalapi
172.120.3.137 overcloud-controller-0.storage.localdomain overcloud-controller-0.storage
172.120.4.152 overcloud-controller-0.storagemgmt.localdomain overcloud-controller-0.storagemgmt
172.120.2.135 overcloud-controller-0.tenant.localdomain overcloud-controller-0.tenant
172.150.100.73 overcloud-controller-0.composable1.localdomain overcloud-controller-0.composable1
fd00:fd00:fd00:8000::1f1 overcloud-controller-0.composable2.localdomain overcloud-controller-0.composable2
192.168.24.14 overcloud-controller-0.ctlplane.localdomain overcloud-controller-0.ctlplane
172.120.1.37 overcloud-controller-1.localdomain overcloud-controller-1
10.0.0.124 overcloud-controller-1.external.localdomain overcloud-controller-1.external
172.120.1.37 overcloud-controller-1.internalapi.localdomain overcloud-controller-1.internalapi
172.120.3.93 overcloud-controller-1.storage.localdomain overcloud-controller-1.storage
172.120.4.83 overcloud-controller-1.storagemgmt.localdomain overcloud-controller-1.storagemgmt
172.120.2.35 overcloud-controller-1.tenant.localdomain overcloud-controller-1.tenant
172.150.100.71 overcloud-controller-1.composable1.localdomain overcloud-controller-1.composable1
fd00:fd00:fd00:8000::49 overcloud-controller-1.composable2.localdomain overcloud-controller-1.composable2
192.168.24.8 overcloud-controller-1.ctlplane.localdomain overcloud-controller-1.ctlplane
172.120.1.94 overcloud-controller-2.localdomain overcloud-controller-2
10.0.0.139 overcloud-controller-2.external.localdomain overcloud-controller-2.external
172.120.1.94 overcloud-controller-2.internalapi.localdomain overcloud-controller-2.internalapi
172.120.3.89 overcloud-controller-2.storage.localdomain overcloud-controller-2.storage
172.120.4.11 overcloud-controller-2.storagemgmt.localdomain overcloud-controller-2.storagemgmt
172.120.2.33 overcloud-controller-2.tenant.localdomain overcloud-controller-2.tenant
172.150.100.101 overcloud-controller-2.composable1.localdomain overcloud-controller-2.composable1
fd00:fd00:fd00:8000::2c2 overcloud-controller-2.composable2.localdomain overcloud-controller-2.composable2
192.168.24.22 overcloud-controller-2.ctlplane.localdomain overcloud-controller-2.ctlplane

172.117.1.40 overcloud-novacompute1-0.localdomain overcloud-novacompute1-0
172.117.2.159 overcloud-novacompute1-0.tenant1.localdomain overcloud-novacompute1-0.tenant1
172.117.3.157 overcloud-novacompute1-0.storage1.localdomain overcloud-novacompute1-0.storage1
172.117.1.40 overcloud-novacompute1-0.internalapi1.localdomain overcloud-novacompute1-0.internalapi1
192.168.24.11 overcloud-novacompute1-0.ctlplane.localdomain overcloud-novacompute1-0.ctlplane
172.117.1.26 overcloud-novacompute1-1.localdomain overcloud-novacompute1-1
172.117.2.124 overcloud-novacompute1-1.tenant1.localdomain overcloud-novacompute1-1.tenant1
172.117.3.193 overcloud-novacompute1-1.storage1.localdomain overcloud-novacompute1-1.storage1
172.117.1.26 overcloud-novacompute1-1.internalapi1.localdomain overcloud-novacompute1-1.internalapi1
192.168.24.21 overcloud-novacompute1-1.ctlplane.localdomain overcloud-novacompute1-1.ctlplane

172.118.1.21 overcloud-novacompute2-0.localdomain overcloud-novacompute2-0
172.118.2.59 overcloud-novacompute2-0.tenant2.localdomain overcloud-novacompute2-0.tenant2
172.118.3.99 overcloud-novacompute2-0.storage2.localdomain overcloud-novacompute2-0.storage2
172.118.1.21 overcloud-novacompute2-0.internalapi2.localdomain overcloud-novacompute2-0.internalapi2
192.168.24.24 overcloud-novacompute2-0.ctlplane.localdomain overcloud-novacompute2-0.ctlplane
172.118.1.16 overcloud-novacompute2-1.localdomain overcloud-novacompute2-1
172.118.2.104 overcloud-novacompute2-1.tenant2.localdomain overcloud-novacompute2-1.tenant2
172.118.3.194 overcloud-novacompute2-1.storage2.localdomain overcloud-novacompute2-1.storage2
172.118.1.16 overcloud-novacompute2-1.internalapi2.localdomain overcloud-novacompute2-1.internalapi2
192.168.24.10 overcloud-novacompute2-1.ctlplane.localdomain overcloud-novacompute2-1.ctlplane

172.119.1.78 overcloud-novacompute3-0.localdomain overcloud-novacompute3-0
172.119.2.89 overcloud-novacompute3-0.tenant3.localdomain overcloud-novacompute3-0.tenant3
172.119.3.24 overcloud-novacompute3-0.storage3.localdomain overcloud-novacompute3-0.storage3
172.119.1.78 overcloud-novacompute3-0.internalapi3.localdomain overcloud-novacompute3-0.internalapi3
192.168.24.13 overcloud-novacompute3-0.ctlplane.localdomain overcloud-novacompute3-0.ctlplane
172.119.1.50 overcloud-novacompute3-1.localdomain overcloud-novacompute3-1
172.119.2.125 overcloud-novacompute3-1.tenant3.localdomain overcloud-novacompute3-1.tenant3
172.119.3.27 overcloud-novacompute3-1.storage3.localdomain overcloud-novacompute3-1.storage3
172.119.1.50 overcloud-novacompute3-1.internalapi3.localdomain overcloud-novacompute3-1.internalapi3
192.168.24.7 overcloud-novacompute3-1.ctlplane.localdomain overcloud-novacompute3-1.ctlplane

172.117.3.61 overcloud-cephstorage1-0.localdomain overcloud-cephstorage1-0
172.117.4.73 overcloud-cephstorage1-0.storagemgmt1.localdomain overcloud-cephstorage1-0.storagemgmt1
172.117.3.61 overcloud-cephstorage1-0.storage1.localdomain overcloud-cephstorage1-0.storage1
192.168.24.9 overcloud-cephstorage1-0.ctlplane.localdomain overcloud-cephstorage1-0.ctlplane
172.117.3.118 overcloud-cephstorage1-1.localdomain overcloud-cephstorage1-1
172.117.4.124 overcloud-cephstorage1-1.storagemgmt1.localdomain overcloud-cephstorage1-1.storagemgmt1
172.117.3.118 overcloud-cephstorage1-1.storage1.localdomain overcloud-cephstorage1-1.storage1
192.168.24.16 overcloud-cephstorage1-1.ctlplane.localdomain overcloud-cephstorage1-1.ctlplane

172.118.3.98 overcloud-cephstorage2-0.localdomain overcloud-cephstorage2-0
172.118.4.193 overcloud-cephstorage2-0.storagemgmt2.localdomain overcloud-cephstorage2-0.storagemgmt2
172.118.3.98 overcloud-cephstorage2-0.storage2.localdomain overcloud-cephstorage2-0.storage2
192.168.24.23 overcloud-cephstorage2-0.ctlplane.localdomain overcloud-cephstorage2-0.ctlplane
172.118.3.140 overcloud-cephstorage2-1.localdomain overcloud-cephstorage2-1
172.118.4.40 overcloud-cephstorage2-1.storagemgmt2.localdomain overcloud-cephstorage2-1.storagemgmt2
172.118.3.140 overcloud-cephstorage2-1.storage2.localdomain overcloud-cephstorage2-1.storage2
192.168.24.20 overcloud-cephstorage2-1.ctlplane.localdomain overcloud-cephstorage2-1.ctlplane

172.119.3.187 overcloud-cephstorage3-0.localdomain overcloud-cephstorage3-0
172.119.4.11 overcloud-cephstorage3-0.storagemgmt3.localdomain overcloud-cephstorage3-0.storagemgmt3
172.119.3.187 overcloud-cephstorage3-0.storage3.localdomain overcloud-cephstorage3-0.storage3
192.168.24.6 overcloud-cephstorage3-0.ctlplane.localdomain overcloud-cephstorage3-0.ctlplane
172.119.3.150 overcloud-cephstorage3-1.localdomain overcloud-cephstorage3-1
172.119.4.100 overcloud-cephstorage3-1.storagemgmt3.localdomain overcloud-cephstorage3-1.storagemgmt3
172.119.3.150 overcloud-cephstorage3-1.storage3.localdomain overcloud-cephstorage3-1.storage3
192.168.24.17 overcloud-cephstorage3-1.ctlplane.localdomain overcloud-cephstorage3-1.ctlplane

# HEAT_HOSTS_END

Comment 4 Alexander Chuzhoy 2019-07-19 18:53:01 UTC
Was able to re-deploy from scratch implementing notes in comment #3.
The /etc/hosts file looks good now.

The reported issue with launching instance persists.
Thanks.

Comment 5 Harald Jensås 2019-07-19 22:44:46 UTC
After adding a hiera override for ``ovn::controller::ovn_encap_ip`` things are looking better.

$ cat templates/extraconfig.yaml
--------------------------------
parameter_defaults:
  ServiceNetMap:
    Compute1HostnameResolveNetwork: internal_api1     <-- It is still correct to add these. We should
    Compute2HostnameResolveNetwork: internal_api2         open doc bug for OSP-13, OSP-14, OSP-15 to add this.
    Compute3HostnameResolveNetwork: internal_api3         (Altough, not having them have worked so far ...)
    CephStorage1HostnameResolveNetwork: storage1
    CephStorage2HostnameResolveNetwork: storage2
    CephStorage3HostnameResolveNetwork: storage3
  Compute1ExtraConfig:
     # neutron::agents::ml2::ovs::local_ip: "%{hiera('tenant1')}"   <-- Removed, as we don't deploy OVS
     nova::my_ip: "%{hiera('internal_api1')}"
     cold_migration_ssh_inbound_addr: "%{hiera('internal_api1')}"
     live_migration_ssh_inbound_addr: "%{hiera('internal_api1')}"
     nova::compute::libvirt::vncserver_listen: "%{hiera('internal_api1')}"
     nova::compute::vncserver_proxyclient_address: "%{hiera('internal_api1')}"
     nova::migration::libvirt::live_migration_inbound_addr: "%{hiera('internal_api1')}"
     tripleo::profile::base::database::mysql::client::mysql_client_bind_address: "%{hiera('internal_api1')}"
     ovn::controller::ovn_encap_ip: "%{hiera('tenant1')}"          <-- Added to override ecapsulation IP for OVN
  Compute2ExtraConfig:
     # neutron::agents::ml2::ovs::local_ip: "%{hiera('tenant2')}"
     nova::my_ip: "%{hiera('internal_api2')}"
     cold_migration_ssh_inbound_addr: "%{hiera('internal_api2')}"
     live_migration_ssh_inbound_addr: "%{hiera('internal_api2')}"
     nova::compute::libvirt::vncserver_listen: "%{hiera('internal_api2')}"
     nova::compute::vncserver_proxyclient_address: "%{hiera('internal_api2')}"
     nova::migration::libvirt::live_migration_inbound_addr: "%{hiera('internal_api2')}"
     tripleo::profile::base::database::mysql::client::mysql_client_bind_address: "%{hiera('internal_api2')}"
     ovn::controller::ovn_encap_ip: "%{hiera('tenant2')}"
  Compute3ExtraConfig:
     # neutron::agents::ml2::ovs::local_ip: "%{hiera('tenant3')}"
     nova::my_ip: "%{hiera('internal_api3')}"
     cold_migration_ssh_inbound_addr: "%{hiera('internal_api3')}"
     live_migration_ssh_inbound_addr: "%{hiera('internal_api3')}"
     nova::compute::libvirt::vncserver_listen: "%{hiera('internal_api3')}"
     nova::compute::vncserver_proxyclient_address: "%{hiera('internal_api3')}"
     nova::migration::libvirt::live_migration_inbound_addr: "%{hiera('internal_api3')}"
     tripleo::profile::base::database::mysql::client::mysql_client_bind_address: "%{hiera('internal_api3')}"
     ovn::controller::ovn_encap_ip: "%{hiera('tenant3')}"
  CephPoolDefaultPgNum: 64
  CephAnsibleExtraConfig: 
    public_network:
      '172.120.3.0/24,172.117.3.0/24,172.118.3.0/24,172.119.3.0/24'
    cluster_network:
      '172.120.4.0/24,172.117.4.0/24,172.118.4.0/24,172.119.4.0/24'


Results:
########

In the ovn-controller.log on compute nodes we no longer see this message repeatedly
-----------------------------------------------------------------------------------

   2019-07-19T21:42:26.064Z|09646|chassis|INFO|Need to specify an encap type and ip

Critical error in /var/log/containers/neutron/ovn-metadata-agent.log is no longer there
---------------------------------------------------------------------------------------

2019-07-18 22:35:46.906 339796 CRITICAL neutron [-] Unhandled error: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=ee6e89cc-9683-4b7c-a7f7-120df9bb6930

We see the OVN agents (Prior to updating hiera override only the agent on the controller nodes was there.)
----------------------------------------------------------------------------------------------------------

(overcloud) [stack@undercloud-0 ~]$ openstack network agent list
+--------------------------------------+----------------------+--------------------------------------+-------------------+-------+-------+-------------------------------+
| ID                                   | Agent Type           | Host                                 | Availability Zone | Alive | State | Binary                        |
+--------------------------------------+----------------------+--------------------------------------+-------------------+-------+-------+-------------------------------+
| 5c289ac7-4485-481c-813f-12233c69a089 | OVN Controller agent | overcloud-controller-1.localdomain   | n/a               | :-)   | UP    | ovn-controller                |
| 8bd5655b-e71d-423f-9a65-5f17fbdadb60 | OVN Controller agent | overcloud-controller-0.localdomain   | n/a               | :-)   | UP    | ovn-controller                |
| e8b2c262-3dd4-4f35-8be0-453a8941b494 | OVN Controller agent | overcloud-controller-2.localdomain   | n/a               | :-)   | UP    | ovn-controller                |
| fbfee020-3d09-498e-976c-eaae1040243b | OVN Controller agent | overcloud-novacompute3-0.localdomain | n/a               | :-)   | UP    | ovn-controller                |
| eb27729a-1073-4103-8fb0-ede8439028e1 | OVN Metadata agent   | overcloud-novacompute3-0.localdomain | n/a               | :-)   | UP    | networking-ovn-metadata-agent |
| ac41c9ca-6b24-4fd0-b978-d1eff8ef3707 | OVN Controller agent | overcloud-novacompute2-0.localdomain | n/a               | :-)   | UP    | ovn-controller                |
| 4cd9659b-3fb1-47fa-a240-fe0ae39d17d2 | OVN Metadata agent   | overcloud-novacompute2-0.localdomain | n/a               | :-)   | UP    | networking-ovn-metadata-agent |
| 94ff8ee2-3d41-43c9-9edb-f821b279513c | OVN Controller agent | overcloud-novacompute1-0.localdomain | n/a               | :-)   | UP    | ovn-controller                |
| 497e0221-fcdb-4811-9d11-5cb38e365312 | OVN Metadata agent   | overcloud-novacompute1-0.localdomain | n/a               | :-)   | UP    | networking-ovn-metadata-agent |
| ed4330d0-c019-4910-a7e4-bc1a44176943 | OVN Controller agent | overcloud-novacompute1-1.localdomain | n/a               | :-)   | UP    | ovn-controller                |
| 548374dd-1e4a-4202-b545-001097b3db09 | OVN Metadata agent   | overcloud-novacompute1-1.localdomain | n/a               | :-)   | UP    | networking-ovn-metadata-agent |
| ebfe981f-9e1d-4b1d-82bc-d0636db148ac | OVN Controller agent | overcloud-novacompute2-1.localdomain | n/a               | :-)   | UP    | ovn-controller                |
| 959b98c9-bfd7-457c-a502-652c9faf26f3 | OVN Metadata agent   | overcloud-novacompute2-1.localdomain | n/a               | :-)   | UP    | networking-ovn-metadata-agent |
| 9d667592-e5ce-40d5-a4db-1f6e451474f1 | OVN Controller agent | overcloud-novacompute3-1.localdomain | n/a               | :-)   | UP    | ovn-controller                |
| adaa1a96-7b3e-4592-8954-5f1ce0438785 | OVN Metadata agent   | overcloud-novacompute3-1.localdomain | n/a               | :-)   | UP    | networking-ovn-metadata-agent |
+--------------------------------------+----------------------+--------------------------------------+-------------------+-------+-------+-------------------------------+



 !! Finally, booting the instance now works !!

Comment 6 Alexander Chuzhoy 2019-07-19 22:57:07 UTC
Switching the bug to doc.

We needed the following directives to successfully launch instance:
(overcloud) [stack@undercloud-0 ~]$ cat /home/stack/templates/extraconfig.yaml                                                                                                        [5/1838]
parameter_defaults:
  ServiceNetMap:
    Compute1HostnameResolveNetwork: internal_api1
    Compute2HostnameResolveNetwork: internal_api2
    Compute3HostnameResolveNetwork: internal_api3
    CephStorage1HostnameResolveNetwork: storage1
    CephStorage2HostnameResolveNetwork: storage2
    CephStorage3HostnameResolveNetwork: storage3
  Compute1ExtraConfig:
     # neutron::agents::ml2::ovs::local_ip: "%{hiera('tenant1')}"
     nova::my_ip: "%{hiera('internal_api1')}"
     cold_migration_ssh_inbound_addr: "%{hiera('internal_api1')}"
     live_migration_ssh_inbound_addr: "%{hiera('internal_api1')}"
     nova::compute::libvirt::vncserver_listen: "%{hiera('internal_api1')}"
     nova::compute::vncserver_proxyclient_address: "%{hiera('internal_api1')}"
     nova::migration::libvirt::live_migration_inbound_addr: "%{hiera('internal_api1')}"
     tripleo::profile::base::database::mysql::client::mysql_client_bind_address: "%{hiera('internal_api1')}"
     ovn::controller::ovn_encap_ip: "%{hiera('tenant1')}"
  Compute2ExtraConfig:
     # neutron::agents::ml2::ovs::local_ip: "%{hiera('tenant2')}"
     nova::my_ip: "%{hiera('internal_api2')}"
     cold_migration_ssh_inbound_addr: "%{hiera('internal_api2')}"
     live_migration_ssh_inbound_addr: "%{hiera('internal_api2')}"
     nova::compute::libvirt::vncserver_listen: "%{hiera('internal_api2')}"
     nova::compute::vncserver_proxyclient_address: "%{hiera('internal_api2')}"
     nova::migration::libvirt::live_migration_inbound_addr: "%{hiera('internal_api2')}"
     tripleo::profile::base::database::mysql::client::mysql_client_bind_address: "%{hiera('internal_api2')}"
     ovn::controller::ovn_encap_ip: "%{hiera('tenant2')}"
  Compute3ExtraConfig:
     # neutron::agents::ml2::ovs::local_ip: "%{hiera('tenant3')}"
     nova::my_ip: "%{hiera('internal_api3')}"
     cold_migration_ssh_inbound_addr: "%{hiera('internal_api3')}"
     live_migration_ssh_inbound_addr: "%{hiera('internal_api3')}"
     nova::compute::libvirt::vncserver_listen: "%{hiera('internal_api3')}"
     nova::compute::vncserver_proxyclient_address: "%{hiera('internal_api3')}"
     nova::migration::libvirt::live_migration_inbound_addr: "%{hiera('internal_api3')}"
     tripleo::profile::base::database::mysql::client::mysql_client_bind_address: "%{hiera('internal_api3')}"
     ovn::controller::ovn_encap_ip: "%{hiera('tenant3')}"


Note the commented out "neutron::agents::ml2::ovs::local_ip" lines and the use of "ovn::controller::ovn_encap_ip:" instead.

Comment 7 Alexander Chuzhoy 2019-07-19 23:02:25 UTC
This is relevant when:
1.deploying OVN with the legacy templates
2. upgrading and migrating to OVN. By default upgrades will stay with OVS.

Comment 8 Alexander Chuzhoy 2019-07-30 22:03:56 UTC
Also note this OC deployment failure when miconfigured ExtraConfig was used:
https://bugzilla.redhat.com/show_bug.cgi?id=1734178

Comment 9 Bob Fournier 2019-08-14 12:20:44 UTC
Doc issue, removing blocker flag.


Note You need to log in before you can comment on or make changes to this bug.