Bug 1700527

Summary: 2nd and 3rd Instance not pingable by FIP on EDGE compute node with Spine Leaf network topology
Product: Red Hat OpenStack Reporter: Yuri Obshansky <yobshans>
Component: openstack-neutronAssignee: James Slagle <jslagle>
Status: CLOSED NOTABUG QA Contact: Yuri Obshansky <yobshans>
Severity: urgent Docs Contact:
Priority: high    
Version: 15.0 (Stein)CC: agurenko, amuller, bdobreli, chrisw, jslagle, ohochman, scohen, twilson
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-29 22:26:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yuri Obshansky 2019-04-16 19:50:36 UTC
Description of problem:
Spine Leaf network topology on virt environment
3 controllers 1 compute - site network
1 compute - dcn1 network
1 compute - dcn2 network

Instance based on RHEL image (7.5/8) started with error in nova-compute.log and not pingable on EDGE compute nodes (leaf1/leaf2).
While Cirros based instance started succesfully and pingable.

2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [req-03f6ff71-dcb0-408b-9fa8-c62f67a4fa7c - - - - -] [instance: 95dc2e39-5b1b-471a-9c95-953697470228] Unable to update binding details for port 3371ecfb-ff57-49cb-b0e6-b7be857e8838: neutronclient.common.exceptions.HostNotCompatibleWithFixedIpsClient: Host site-undercloud-0.localdomain is not connected to a segment where the existing fixed_ips on port 3371ecfb-ff57-49cb-b0e6-b7be857e8838 will function given the routed network topology.
Neutron server returns request_ids: ['req-a203d9ad-cc08-49de-9068-cc4708902d20']
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228] Traceback (most recent call last):
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 3365, in _update_port_binding_for_instance
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     neutron.update_port(port_id, {'port': updates})
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 127, in wrapper
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     ret = obj(*args, **kwargs)
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 808, in update_port
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     revision_number=revision_number)
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 127, in wrapper
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     ret = obj(*args, **kwargs)
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 2389, in _update_resource
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     return self.put(path, **kwargs)
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 127, in wrapper
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     ret = obj(*args, **kwargs)
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 363, in put
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     headers=headers, params=params)
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 127, in wrapper
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     ret = obj(*args, **kwargs)
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     headers=headers, params=params)
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 127, in wrapper
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     ret = obj(*args, **kwargs)
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     self._handle_fault_response(status_code, replybody, resp)
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 127, in wrapper
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     ret = obj(*args, **kwargs)
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     exception_handler_v20(status_code, error_body)
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     request_ids=request_ids)
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228] neutronclient.common.exceptions.HostNotCompatibleWithFixedIpsClient: Host site-undercloud-0.localdomain is not connected to a segment where the existing fixed_ips on port 3371ecfb-ff57-49cb-b0e6-b7be857e8838 will function given the routed network topology.
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228] Neutron server returns request_ids: ['req-a203d9ad-cc08-49de-9068-cc4708902d20']
2019-04-16 17:42:01.686 8 ERROR nova.network.neutronv2.api [instance: 95dc2e39-5b1b-471a-9c95-953697470228] 
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [req-03f6ff71-dcb0-408b-9fa8-c62f67a4fa7c - - - - -] [instance: 95dc2e39-5b1b-471a-9c95-953697470228] An error occurred while refreshing the network cache.: neutronclient.common.exceptions.HostNotCompatibleWithFixedIpsClient: Host site-undercloud-0.localdomain is not connected to a segment where the existing fixed_ips on port 3371ecfb-ff57-49cb-b0e6-b7be857e8838 will function given the routed network topology.
Neutron server returns request_ids: ['req-a203d9ad-cc08-49de-9068-cc4708902d20']
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228] Traceback (most recent call last):
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 7286, in _heal_instance_info_cache
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     context, instance, self.host)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 3243, in setup_instance_network_on_host
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     migration)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 3370, in _update_port_binding_for_instance
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     port_id, instance=instance)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     self.force_reraise()
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     six.reraise(self.type_, self.value, self.tb)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     raise value
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 3365, in _update_port_binding_for_instance
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     neutron.update_port(port_id, {'port': updates})
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 127, in wrapper
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     ret = obj(*args, **kwargs)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 808, in update_port
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     revision_number=revision_number)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 127, in wrapper
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     ret = obj(*args, **kwargs)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 2389, in _update_resource
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     return self.put(path, **kwargs)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 127, in wrapper
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     ret = obj(*args, **kwargs)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 363, in put
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     headers=headers, params=params)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 127, in wrapper
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     ret = obj(*args, **kwargs)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     headers=headers, params=params)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 127, in wrapper
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     ret = obj(*args, **kwargs)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     self._handle_fault_response(status_code, replybody, resp)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py", line 127, in wrapper
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     ret = obj(*args, **kwargs)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     exception_handler_v20(status_code, error_body)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228]     request_ids=request_ids)
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228] neutronclient.common.exceptions.HostNotCompatibleWithFixedIpsClient: Host site-undercloud-0.localdomain is not connected to a segment where the existing fixed_ips on port 3371ecfb-ff57-49cb-b0e6-b7be857e8838 will function given the routed network topology.
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228] Neutron server returns request_ids: ['req-a203d9ad-cc08-49de-9068-cc4708902d20']
2019-04-16 17:42:01.687 8 ERROR nova.compute.manager [instance: 95dc2e39-5b1b-471a-9c95-953697470228] 

Version-Release number of selected component (if applicable):
RHOS_TRUNK-15.0-RHEL-8-20190403.n.3

(overcloud) [stack@site-undercloud-0 ~]$ rpm -qa |grep neutron
python3-neutron-lib-1.25.0-0.20190312185238.fc2a810.el8ost.noarch
puppet-neutron-14.4.1-0.20190329210334.472f32e.el8ost.noarch
python3-neutronclient-6.12.0-0.20190312100012.680b417.el8ost.noarch
(overcloud) [stack@site-undercloud-0 ~]$ rpm -qa |grep nova
python3-novaclient-13.0.0-0.20190312082821.62bf880.el8ost.noarch
puppet-nova-14.4.1-0.20190329030330.0a87c63.el8ost.noarch

How reproducible:


Steps to Reproduce:
1. Deploy OSP 15 environment with Spine Leaf network topology
which has 1 Site and 2 DCN networks
2. 
$ openstack network create --share --external --provider-network-type flat --provider-physical-network datacentre public
$ openstack subnet create --network public --subnet-range 10.0.0.10/24 --no-dhcp public-subnet
$ openstack network create --share --provider-network-type geneve private
$ openstack subnet create private-subnet --subnet-range 192.0.2.0/24 --network private
$ openstack router create default-router
$ openstack router add subnet default-router private-subnet
$ openstack router set --external-gateway public default-router
$ openstack floating ip create public
$ openstack server create --flavor m1.tiny --image cirros --network private vm1
$ openstack server add floating ip vm1 10.0.0.10
$ openstack floating ip create public
$ openstack server create --flavor m1.small --image rhel --network private vm2
$ openstack server add floating ip vm2 10.0.0.148
3.

Actual results:
Instance vm1 based on cirros image created, added floating IP and pingable
| 6271015e-2a51-4498-ac10-921594900c17 | vm1  | ACTIVE | private=192.0.2.196, 10.0.0.10  | cirros | m1.tiny  |
(overcloud) [stack@site-undercloud-0 ~]$ ping 10.0.0.10
PING 10.0.0.10 (10.0.0.10) 56(84) bytes of data.
64 bytes from 10.0.0.10: icmp_seq=1 ttl=55 time=155 ms

Instance vm2 based on RHEL 7.5 image created, added IP and NOT pingable
| cd4763a0-5dc8-489d-9161-3ad760811b3c | vm2  | ACTIVE | private=192.0.2.163, 10.0.0.148 | rhel   | m1.small |
(overcloud) [stack@site-undercloud-0 ~]$ ping 10.0.0.148
PING 10.0.0.148 (10.0.0.148) 56(84) bytes of data.
--- 10.0.0.148 ping statistics ---
19 packets transmitted, 0 received, 100% packet loss, time 481ms

Expected results:


Additional info:

Comment 1 Yuri Obshansky 2019-04-17 20:38:36 UTC
Actually, after short investigation I found that only 2nd, 3th instance has problem to be pingable by FIP 
and it doesn't matter which image used for create instance cirros/rhel.
One more error from neutron server.log 
2019-04-16 00:01:56.688 24 ERROR neutron.plugins.ml2.managers [req-6c714175-2afc-47ce-9df0-8df280d34688 08a70f2cde8e42fb9eaded69982e2a36 15a22b489b464678a8a8e2bb72ee2d65 - default default] Failed to bind port 92366bbe-79de-4032-bc6a-e72c3b8b3ae8 on host site-undercloud-0.localdomain for vnic_type baremetal using segments [{'id': 'b6b23fd7-7bdd-4169-aaae-284215f0b268', 'network_type': 'flat', 'physical_network': 'ctlplane', 'segmentation_id': None, 'network_id': 'ed0f7ac5-fe24-4390-aa5f-da216aed9c50'}, {'id': '5c419847-095b-44f5-a3fd-40402cc3ce51', 'network_type': 'flat', 'physical_network': 'leaf1', 'segmentation_id': None, 'network_id': 'ed0f7ac5-fe24-4390-aa5f-da216aed9c50'}, {'id': '9fa1784a-8373-4e5d-8845-0c5e745d9c09', 'network_type': 'flat', 'physical_network': 'leaf2', 'segmentation_id': None, 'network_id': 'ed0f7ac5-fe24-4390-aa5f-da216aed9c50'}]
 
Bug will be rename.

Yuri

Comment 4 Yuri Obshansky 2019-04-29 22:26:30 UTC
Looks like issue with binding port of Controller on undercloud
2019-04-29 21:59:37.296 25 ERROR neutron.plugins.ml2.managers [req-de7d48d3-a59e-4f84-a3ab-0f6a2b15d367 0987497e65dd434eb63256f7e2f78420 d120123a294c4e5fbfe597e1d93a75a3 - default default] Failed to bind port b8de870d-eed6-4277-b60e-b156a4b9d899 on host site-undercloud-0.localdomain for vnic_type baremetal using segments [{'id': 'c647af7f-b3bd-4760-8327-99c85c8eb878', 'network_type': 'flat', 'physical_network': 'ctlplane', 'segmentation_id': None, 'network_id': '5181539c-01e9-4e67-9c8a-44233c14ab30'}, {'id': 'ba9ee18a-c9d8-4363-9f80-6e547ed01439', 'network_type': 'flat', 'physical_network': 'leaf1', 'segmentation_id': None, 'network_id': '5181539c-01e9-4e67-9c8a-44233c14ab30'}, {'id': '8a12209a-54d9-4bf1-86ee-4dbb2a6cb436', 'network_type': 'flat', 'physical_network': 'leaf2', 'segmentation_id': None, 'network_id': '5181539c-01e9-4e67-9c8a-44233c14ab30'}]

(undercloud) [stack@site-undercloud-0 ~]$ openstack port list |grep b8de870d-eed6-4277-b60e-b156a4b9d899
| b8de870d-eed6-4277-b60e-b156a4b9d899 | Controller0-port-0                  | 52:54:00:0b:b6:a8 | ip_address='192.168.24.31', subnet_id='54276d9f-a820-4820-9af1-89585536eade' | DOWN   |

I opened new 
https://bugzilla.redhat.com/show_bug.cgi?id=1704490