Bug 1892000

Summary:	Ironic nodes are unable to get a metadata service with OVN while VMs can get it fine on the same network
Product:	Red Hat OpenStack	Reporter:	Chris Janiszewski <cjanisze>
Component:	python-networking-ovn	Assignee:	Lucas Alvares Gomes <lmartins>
Status:	CLOSED DUPLICATE	QA Contact:	Eran Kuris <ekuris>
Severity:	high	Docs Contact:
Priority:	medium
Version:	16.1 (Train)	CC:	apevec, asimonel, emeroglu, jlibosva, lhh, lmartins, majopela, racedoro, scohen, ushkalim
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-06-20 14:13:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Chris Janiszewski 2020-10-27 20:36:42 UTC

Description of problem:
I am running a use case of deploying workload on a mix of VM and BM using the same automation (OCP IPI). The OS deployed on VMs are able to access metadata service where BM on the same network cannot.
The issue seems to be related to default routing for metadata server, since I can workaround the issue by injecting the static route that matches what's being used on VMs.

VM instance:
[core@ocp-hybrid-57svt-master-0 ~]$ curl http://169.254.169.254/openstack
2012-08-10
2013-04-04
2013-10-17
2015-10-15
2016-06-30
2016-10-06
2017-02-22
2018-08-27
[core@ocp-hybrid-57svt-master-0 ~]$ ip route
default via 10.9.67.65 dev ens3 proto dhcp metric 100
default via 10.9.65.1 dev ens4 proto dhcp metric 101
10.9.65.0/27 dev ens4 proto kernel scope link src 10.9.65.25 metric 101
10.9.67.64/28 dev ens3 proto kernel scope link src 10.9.67.72 metric 100
10.128.0.0/14 dev tun0 scope link
169.254.169.254 via 10.9.67.68 dev ens3 proto dhcp metric 100
169.254.169.254 via 10.9.65.9 dev ens4 proto dhcp metric 101
172.30.0.0/16 dev tun0


BM instance:
[root@host-10-9-67-75 ~]# curl http://169.254.169.254/openstack
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
[root@host-10-9-67-75 ~]# ip route
default via 10.9.67.65 dev ens1f0 proto dhcp metric 101
default via 10.9.65.1 dev ens1f1 proto dhcp metric 102
10.9.65.0/27 dev ens1f1 proto kernel scope link src 10.9.65.16 metric 102
10.9.67.64/28 dev ens1f0 proto kernel scope link src 10.9.67.75 metric 101
169.254.95.0/24 dev enp0s20u1u5 proto kernel scope link src 169.254.95.120 metric 100
169.254.169.254 via 10.9.67.70 dev ens1f0 proto dhcp metric 101
169.254.169.254 via 10.9.65.10 dev ens1f1 proto dhcp metric 102


hacky workaround:
[root@host-10-9-67-75 ~]# ip route a 169.254.169.254 via 10.9.67.68 dev ens1f0 proto dhcp metric 100
[root@host-10-9-67-75 ~]# curl http://169.254.169.254/openstack
2012-08-10
2013-04-04
2013-10-17
2015-10-15
2016-06-30
2016-10-06
2017-02-22
2018-08-27

and then too keep this persistent I have injected this route to a subnet itself
(ocp-hybrid) [stack@undercloud-hextupleo ~]$ openstack subnet show ocp-hybrid-tenant-net1-subnet --column host_routes
+-------------+--------------------------------------------------------+
| Field       | Value                                                  |
+-------------+--------------------------------------------------------+
| host_routes | destination='169.254.169.254/32', gateway='10.9.67.68' |
+-------------+--------------------------------------------------------+


Here is the info about the individual ports that are used for metadata routing by default (sorry for a lot for text):
(ocp-hybrid) [stack@undercloud-hextupleo ~]$ openstack port list | grep 10.9.67.                                                                                                                            
| 133696de-4a8e-4c61-85ad-3a0060fac658 |                                 | fa:16:3e:fa:c2:d8 | ip_address='10.9.67.70', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72'  | ACTIVE |                        
| 27717a7e-0f1c-4850-8f4d-fecea9eccf95 | ocp-hybrid-fdh6x-api-port       | fa:16:3e:1a:a9:25 | ip_address='10.9.67.78', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72'  | DOWN   |                        
| 292164a9-e757-49ab-a58f-e83896a8b70b | ocp-hybrid-fdh6x-worker-0-fg7xs | 00:0e:1e:aa:e2:e0 | ip_address='10.9.67.72', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72'  | ACTIVE |                        
| 441fa0d9-c0ba-4cf8-bf36-348651e9c5e7 |                                 | fa:16:3e:48:90:b2 | ip_address='10.9.67.146', subnet_id='fc755cbc-2293-42da-8a6d-7b1fd44cbbcb' | DOWN   |                        
| 56ec67f8-44bb-4c16-b529-06025dbb0594 | ocp-hybrid-fdh6x-master-port-1  | fa:16:3e:b1:07:8e | ip_address='10.9.67.75', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72'  | ACTIVE |                        
| 5fa52924-1fe0-4a16-be31-856990d2c442 | ocp-hybrid-fdh6x-ingress-port   | fa:16:3e:13:d0:56 | ip_address='10.9.67.77', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72'  | DOWN   |                        
| 6e037c51-4952-435a-baf8-9cb1cc4261eb |                                 | fa:16:3e:62:3e:45 | ip_address='10.9.67.149', subnet_id='fc755cbc-2293-42da-8a6d-7b1fd44cbbcb' | ACTIVE |                        
| 758edc9c-25c9-4bd2-b2bb-e427b8ae2969 | ocp-hybrid-fdh6x-master-port-2  | fa:16:3e:36:2f:77 | ip_address='10.9.67.73', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72'  | ACTIVE |                        
| 7e6bd603-37f5-4a77-9e3f-9ee47a91b95b |                                 | fa:16:3e:cf:49:6d | ip_address='10.9.67.69', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72'  | ACTIVE |                        
| 7f872127-ec17-4f3e-b1fa-6175f97e3b8f |                                 | fa:16:3e:89:67:92 | ip_address='10.9.67.68', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72'  | DOWN   |                        
| 8832aedb-0ff4-431a-8dd9-8de075b975d2 |                                 | fa:16:3e:a5:54:4d | ip_address='10.9.67.147', subnet_id='fc755cbc-2293-42da-8a6d-7b1fd44cbbcb' | ACTIVE |                        
| 9f89d798-74a2-4f7d-86ce-f46aabea1a5d | ocp-hybrid-fdh6x-worker-0-nks6k | 00:0e:1e:aa:d8:f0 | ip_address='10.9.67.76', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72'  | ACTIVE |                        
| c90653bd-25da-4c1e-9f20-f71d97ae5ab9 | ocp-hybrid-fdh6x-master-port-0  | fa:16:3e:2f:03:94 | ip_address='10.9.67.74', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72'  | ACTIVE |                        
| dee04cdc-186a-4b1d-a9d8-8ea8a64bd60b |                                 | fa:16:3e:3d:33:81 | ip_address='10.9.67.71', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72'  | ACTIVE |
| ea28e6fc-4a67-4734-ae9b-83503b4c2efd |                                 | fa:16:3e:22:3a:e3 | ip_address='10.9.67.148', subnet_id='fc755cbc-2293-42da-8a6d-7b1fd44cbbcb' | ACTIVE |
(ocp-hybrid) [stack@undercloud-hextupleo ~]$ openstack port show 133696de-4a8e-4c61-85ad-3a0060fac658
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                                                       |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | UP                                                                                                                                                                          |
| allowed_address_pairs   |                                                                                                                                                                             |
| binding_host_id         | hextupleo-controller-1.localdomain                                                                                                                                          |
| binding_profile         |                                                                                                                                                                             |
| binding_vif_details     | port_filter='True'                                                                                                                                                          |
| binding_vif_type        | ovs                                                                                                                                                                         |
| binding_vnic_type       | normal                                                                                                                                                                      |
| created_at              | 2020-10-27T15:11:39Z                                                                                                                                                        |
| data_plane_status       | None                                                                                                                                                                        |
| description             |                                                                                                                                                                             |
| device_id               | dhcp059495c4-03c9-538c-a9d1-a81ee090a79a-9668e989-10fb-409d-9140-2621fd0cb00b                                                                                               |
| device_owner            | network:dhcp                                                                                                                                                                |
| dns_assignment          | fqdn='host-10-9-67-70.openstacklocal.', hostname='host-10-9-67-70', ip_address='10.9.67.70'                                                                                 |
| dns_domain              | None                                                                                                                                                                        |
| dns_name                |                                                                                                                                                                             |
| extra_dhcp_opts         |                                                                                                                                                                             |
| fixed_ips               | ip_address='10.9.67.70', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72'                                                                                                   |
| id                      | 133696de-4a8e-4c61-85ad-3a0060fac658                                                                                                                                        |
| location                | cloud='', project.domain_id=, project.domain_name='Default', project.id='2a851f52aef24185805472d044dc2512', project.name='ocp-hybrid-vm-bm', region_name='regionOne', zone= |
| mac_address             | fa:16:3e:fa:c2:d8                                                                                                                                                           |
| name                    |                                                                                                                                                                             |
| network_id              | 9668e989-10fb-409d-9140-2621fd0cb00b                                                                                                                                        |
| port_security_enabled   | False                                                                                                                                                                       |
| project_id              | 2a851f52aef24185805472d044dc2512                                                                                                                                            |
| propagate_uplink_status | None                                                                                                                                                                        |
| qos_policy_id           | None                                                                                                                                                                        |
| resource_request        | None                                                                                                                                                                        |
| revision_number         | 3                                                                                                                                                                           |
| security_group_ids      |                                                                                                                                                                             |
| status                  | ACTIVE                                                                                                                                                                      |
| tags                    |                                                                                                                                                                             |
| trunk_details           | None                                                                                                                                                                        |
| updated_at              | 2020-10-27T15:11:57Z                                                                                                                                                        |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(ocp-hybrid) [stack@undercloud-hextupleo ~]$ openstack port show 7f872127-ec17-4f3e-b1fa-6175f97e3b8f
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                                                       |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | UP                                                                                                                                                                          |
| allowed_address_pairs   |                                                                                                                                                                             |
| binding_host_id         |                                                                                                                                                                             |
| binding_profile         |                                                                                                                                                                             |
| binding_vif_details     |                                                                                                                                                                             |
| binding_vif_type        | unbound                                                                                                                                                                     |
| binding_vnic_type       | normal                                                                                                                                                                      |
| created_at              | 2020-10-27T15:11:33Z                                                                                                                                                        |
| data_plane_status       | None                                                                                                                                                                        |
| description             |                                                                                                                                                                             |
| device_id               | ovnmeta-9668e989-10fb-409d-9140-2621fd0cb00b                                                                                                                                |
| device_owner            | network:dhcp                                                                                                                                                                |
| dns_assignment          | fqdn='host-10-9-67-68.openstacklocal.', hostname='host-10-9-67-68', ip_address='10.9.67.68'                                                                                 |
| dns_domain              | None                                                                                                                                                                        |
| dns_name                |                                                                                                                                                                             |
| extra_dhcp_opts         |                                                                                                                                                                             |
| fixed_ips               | ip_address='10.9.67.68', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72'                                                                                                   |
| id                      | 7f872127-ec17-4f3e-b1fa-6175f97e3b8f                                                                                                                                        |
| location                | cloud='', project.domain_id=, project.domain_name='Default', project.id='2a851f52aef24185805472d044dc2512', project.name='ocp-hybrid-vm-bm', region_name='regionOne', zone= |
| mac_address             | fa:16:3e:89:67:92                                                                                                                                                           |
| name                    |                                                                                                                                                                             |
| network_id              | 9668e989-10fb-409d-9140-2621fd0cb00b                                                                                                                                        |
| port_security_enabled   | False                                                                                                                                                                       |
| project_id              | 2a851f52aef24185805472d044dc2512                                                                                                                                            |
| propagate_uplink_status | None                                                                                                                                                                        |
| qos_policy_id           | None                                                                                                                                                                        |
| resource_request        | None                                                                                                                                                                        |
| revision_number         | 2                                                                                                                                                                           |
| security_group_ids      |                                                                                                                                                                             |
| status                  | DOWN                                                                                                                                                                        |
| tags                    |                                                                                                                                                                             |
| trunk_details           | None                                                                                                                                                                        |
| updated_at              | 2020-10-27T15:11:38Z                                                                                                                                                        |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+




 

Version-Release number of selected component (if applicable):
OSP16.1.1

How reproducible:
everytime

Steps to Reproduce:
1. deploy BM on provider network
2. verify if the metadata service is available
3.

Actual results:


Expected results:


Additional info:

parameter_defaults:
  NeutronEnableIsolatedMetadata: True

^^ this has been configuired

Comment 13 Lucas Alvares Gomes 2022-06-20 14:13:08 UTC


*** This bug has been marked as a duplicate of bug 2033953 ***