Description of problem: I am running a use case of deploying workload on a mix of VM and BM using the same automation (OCP IPI). The OS deployed on VMs are able to access metadata service where BM on the same network cannot. The issue seems to be related to default routing for metadata server, since I can workaround the issue by injecting the static route that matches what's being used on VMs. VM instance: [core@ocp-hybrid-57svt-master-0 ~]$ curl http://169.254.169.254/openstack 2012-08-10 2013-04-04 2013-10-17 2015-10-15 2016-06-30 2016-10-06 2017-02-22 2018-08-27 [core@ocp-hybrid-57svt-master-0 ~]$ ip route default via 10.9.67.65 dev ens3 proto dhcp metric 100 default via 10.9.65.1 dev ens4 proto dhcp metric 101 10.9.65.0/27 dev ens4 proto kernel scope link src 10.9.65.25 metric 101 10.9.67.64/28 dev ens3 proto kernel scope link src 10.9.67.72 metric 100 10.128.0.0/14 dev tun0 scope link 169.254.169.254 via 10.9.67.68 dev ens3 proto dhcp metric 100 169.254.169.254 via 10.9.65.9 dev ens4 proto dhcp metric 101 172.30.0.0/16 dev tun0 BM instance: [root@host-10-9-67-75 ~]# curl http://169.254.169.254/openstack <html><body><h1>503 Service Unavailable</h1> No server is available to handle this request. </body></html> [root@host-10-9-67-75 ~]# ip route default via 10.9.67.65 dev ens1f0 proto dhcp metric 101 default via 10.9.65.1 dev ens1f1 proto dhcp metric 102 10.9.65.0/27 dev ens1f1 proto kernel scope link src 10.9.65.16 metric 102 10.9.67.64/28 dev ens1f0 proto kernel scope link src 10.9.67.75 metric 101 169.254.95.0/24 dev enp0s20u1u5 proto kernel scope link src 169.254.95.120 metric 100 169.254.169.254 via 10.9.67.70 dev ens1f0 proto dhcp metric 101 169.254.169.254 via 10.9.65.10 dev ens1f1 proto dhcp metric 102 hacky workaround: [root@host-10-9-67-75 ~]# ip route a 169.254.169.254 via 10.9.67.68 dev ens1f0 proto dhcp metric 100 [root@host-10-9-67-75 ~]# curl http://169.254.169.254/openstack 2012-08-10 2013-04-04 2013-10-17 2015-10-15 2016-06-30 2016-10-06 2017-02-22 2018-08-27 and then too keep this persistent I have injected this route to a subnet itself (ocp-hybrid) [stack@undercloud-hextupleo ~]$ openstack subnet show ocp-hybrid-tenant-net1-subnet --column host_routes +-------------+--------------------------------------------------------+ | Field | Value | +-------------+--------------------------------------------------------+ | host_routes | destination='169.254.169.254/32', gateway='10.9.67.68' | +-------------+--------------------------------------------------------+ Here is the info about the individual ports that are used for metadata routing by default (sorry for a lot for text): (ocp-hybrid) [stack@undercloud-hextupleo ~]$ openstack port list | grep 10.9.67. | 133696de-4a8e-4c61-85ad-3a0060fac658 | | fa:16:3e:fa:c2:d8 | ip_address='10.9.67.70', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72' | ACTIVE | | 27717a7e-0f1c-4850-8f4d-fecea9eccf95 | ocp-hybrid-fdh6x-api-port | fa:16:3e:1a:a9:25 | ip_address='10.9.67.78', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72' | DOWN | | 292164a9-e757-49ab-a58f-e83896a8b70b | ocp-hybrid-fdh6x-worker-0-fg7xs | 00:0e:1e:aa:e2:e0 | ip_address='10.9.67.72', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72' | ACTIVE | | 441fa0d9-c0ba-4cf8-bf36-348651e9c5e7 | | fa:16:3e:48:90:b2 | ip_address='10.9.67.146', subnet_id='fc755cbc-2293-42da-8a6d-7b1fd44cbbcb' | DOWN | | 56ec67f8-44bb-4c16-b529-06025dbb0594 | ocp-hybrid-fdh6x-master-port-1 | fa:16:3e:b1:07:8e | ip_address='10.9.67.75', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72' | ACTIVE | | 5fa52924-1fe0-4a16-be31-856990d2c442 | ocp-hybrid-fdh6x-ingress-port | fa:16:3e:13:d0:56 | ip_address='10.9.67.77', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72' | DOWN | | 6e037c51-4952-435a-baf8-9cb1cc4261eb | | fa:16:3e:62:3e:45 | ip_address='10.9.67.149', subnet_id='fc755cbc-2293-42da-8a6d-7b1fd44cbbcb' | ACTIVE | | 758edc9c-25c9-4bd2-b2bb-e427b8ae2969 | ocp-hybrid-fdh6x-master-port-2 | fa:16:3e:36:2f:77 | ip_address='10.9.67.73', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72' | ACTIVE | | 7e6bd603-37f5-4a77-9e3f-9ee47a91b95b | | fa:16:3e:cf:49:6d | ip_address='10.9.67.69', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72' | ACTIVE | | 7f872127-ec17-4f3e-b1fa-6175f97e3b8f | | fa:16:3e:89:67:92 | ip_address='10.9.67.68', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72' | DOWN | | 8832aedb-0ff4-431a-8dd9-8de075b975d2 | | fa:16:3e:a5:54:4d | ip_address='10.9.67.147', subnet_id='fc755cbc-2293-42da-8a6d-7b1fd44cbbcb' | ACTIVE | | 9f89d798-74a2-4f7d-86ce-f46aabea1a5d | ocp-hybrid-fdh6x-worker-0-nks6k | 00:0e:1e:aa:d8:f0 | ip_address='10.9.67.76', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72' | ACTIVE | | c90653bd-25da-4c1e-9f20-f71d97ae5ab9 | ocp-hybrid-fdh6x-master-port-0 | fa:16:3e:2f:03:94 | ip_address='10.9.67.74', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72' | ACTIVE | | dee04cdc-186a-4b1d-a9d8-8ea8a64bd60b | | fa:16:3e:3d:33:81 | ip_address='10.9.67.71', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72' | ACTIVE | | ea28e6fc-4a67-4734-ae9b-83503b4c2efd | | fa:16:3e:22:3a:e3 | ip_address='10.9.67.148', subnet_id='fc755cbc-2293-42da-8a6d-7b1fd44cbbcb' | ACTIVE | (ocp-hybrid) [stack@undercloud-hextupleo ~]$ openstack port show 133696de-4a8e-4c61-85ad-3a0060fac658 +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | hextupleo-controller-1.localdomain | | binding_profile | | | binding_vif_details | port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | normal | | created_at | 2020-10-27T15:11:39Z | | data_plane_status | None | | description | | | device_id | dhcp059495c4-03c9-538c-a9d1-a81ee090a79a-9668e989-10fb-409d-9140-2621fd0cb00b | | device_owner | network:dhcp | | dns_assignment | fqdn='host-10-9-67-70.openstacklocal.', hostname='host-10-9-67-70', ip_address='10.9.67.70' | | dns_domain | None | | dns_name | | | extra_dhcp_opts | | | fixed_ips | ip_address='10.9.67.70', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72' | | id | 133696de-4a8e-4c61-85ad-3a0060fac658 | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='2a851f52aef24185805472d044dc2512', project.name='ocp-hybrid-vm-bm', region_name='regionOne', zone= | | mac_address | fa:16:3e:fa:c2:d8 | | name | | | network_id | 9668e989-10fb-409d-9140-2621fd0cb00b | | port_security_enabled | False | | project_id | 2a851f52aef24185805472d044dc2512 | | propagate_uplink_status | None | | qos_policy_id | None | | resource_request | None | | revision_number | 3 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2020-10-27T15:11:57Z | +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (ocp-hybrid) [stack@undercloud-hextupleo ~]$ openstack port show 7f872127-ec17-4f3e-b1fa-6175f97e3b8f +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | | | binding_profile | | | binding_vif_details | | | binding_vif_type | unbound | | binding_vnic_type | normal | | created_at | 2020-10-27T15:11:33Z | | data_plane_status | None | | description | | | device_id | ovnmeta-9668e989-10fb-409d-9140-2621fd0cb00b | | device_owner | network:dhcp | | dns_assignment | fqdn='host-10-9-67-68.openstacklocal.', hostname='host-10-9-67-68', ip_address='10.9.67.68' | | dns_domain | None | | dns_name | | | extra_dhcp_opts | | | fixed_ips | ip_address='10.9.67.68', subnet_id='ed415e64-09ae-43ac-8129-03fadbc5ee72' | | id | 7f872127-ec17-4f3e-b1fa-6175f97e3b8f | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='2a851f52aef24185805472d044dc2512', project.name='ocp-hybrid-vm-bm', region_name='regionOne', zone= | | mac_address | fa:16:3e:89:67:92 | | name | | | network_id | 9668e989-10fb-409d-9140-2621fd0cb00b | | port_security_enabled | False | | project_id | 2a851f52aef24185805472d044dc2512 | | propagate_uplink_status | None | | qos_policy_id | None | | resource_request | None | | revision_number | 2 | | security_group_ids | | | status | DOWN | | tags | | | trunk_details | None | | updated_at | 2020-10-27T15:11:38Z | +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ Version-Release number of selected component (if applicable): OSP16.1.1 How reproducible: everytime Steps to Reproduce: 1. deploy BM on provider network 2. verify if the metadata service is available 3. Actual results: Expected results: Additional info: parameter_defaults: NeutronEnableIsolatedMetadata: True ^^ this has been configuired
*** This bug has been marked as a duplicate of bug 2033953 ***