Bug 2066406

Description of problem:

Since https://bugzilla.redhat.com/show_bug.cgi?id=2041893 fix landed, ipi proxy installation with Kuryr is working, but the destroy cluster is not.

We are creating a installer_VM with two nics, one connected to the installer_host-subnet with external access, and one connected to restricted_subnet without external access.
Then we are installing a proxy on the installer_host and triggering the IPI openshift installation with Kuryr with BYON using the restricted_subnet. CNO is creating a router, because the router that is connected to the installer_host is not visible for the project and it is not connected to the restricted_subnet.

The installation is OK, but the destroy is stuck at this point:

ERROR {"NeutronError": {"type": "RouterInUse", "message": "Router 5af47047-3abc-44c6-82b6-d97e0fe5bca2 still has ports", "detail": ""}}                                                      
DEBUG Exiting deleting openstack routers

The stuck resources that are:

(shiftstack) [cloud-user@installer-host ~]$ openstack router show ostest-sk7dd-external-router
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+                                          
| Field                   | Value                                                                                                                                                                 |                                          
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+                                          
| admin_state_up          | UP                                                                                                                                                                    |                                          
| availability_zone_hints |                                                                                                                                                                       |                                          
| availability_zones      |                                                                                                                                                                       |                                          
| created_at              | 2022-03-21T12:29:06Z                                                                                                                                                  |                                          
| description             |                                                                                                                                                                       |                                          
| external_gateway_info   | null                                                                                                                                                                  |                                          
| flavor_id               | None                                                                                                                                                                  |                                          
| id                      | 5af47047-3abc-44c6-82b6-d97e0fe5bca2                                                                                                                                  |                                          
| interfaces_info         | [{"port_id": "16015b54-dd09-425c-8982-36d440dd224a", "ip_address": "172.16.0.1", "subnet_id": "878a7964-2ae6-452d-8084-08924dafa9d7"}]                                |                                          
| location                | cloud='', project.domain_id=, project.domain_name='Default', project.id='fc02d48e173c411b9bb6dff1fd65265a', project.name='shiftstack', region_name='regionOne', zone= |                                          
| name                    | ostest-sk7dd-external-router                                                                                                                                          |                                          
| project_id              | fc02d48e173c411b9bb6dff1fd65265a                                                                                                                                      |                                          
| revision_number         | 160                                                                                                                                                                   |                                          
| routes                  |                                                                                                                                                                       |                                          
| status                  | ACTIVE                                                                                                                                                                |                                          
| tags                    | openshiftClusterID=ostest-sk7dd                                                                                                                                       |                                          
| updated_at              | 2022-03-21T16:27:01Z                                                                                                                                                  |                                          
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+                                    

(shiftstack) [cloud-user@installer-host ~]$ openstack port show 16015b54-dd09-425c-8982-36d440dd224a
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                                                 |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | UP                                                                                                                                                                    |
| allowed_address_pairs   |                                                                                                                                                                       |
| binding_host_id         | None                                                                                                                                                                  |
| binding_profile         | None                                                                                                                                                                  |
| binding_vif_details     | None                                                                                                                                                                  |
| binding_vif_type        | None                                                                                                                                                                  |
| binding_vnic_type       | normal                                                                                                                                                                |
| created_at              | 2022-03-21T12:29:07Z                                                                                                                                                  |
| data_plane_status       | None                                                                                                                                                                  |
| description             |                                                                                                                                                                       |
| device_id               | 5af47047-3abc-44c6-82b6-d97e0fe5bca2                                                                                                                                  |
| device_owner            | network:router_interface                                                                                                                                              |
| dns_assignment          | fqdn='host-172-16-0-1.shiftstack.com.', hostname='host-172-16-0-1', ip_address='172.16.0.1'                                                                           |
| dns_domain              |                                                                                                                                                                       |
| dns_name                |                                                                                                                                                                       |
| extra_dhcp_opts         |                                                                                                                                                                       |
| fixed_ips               | ip_address='172.16.0.1', subnet_id='878a7964-2ae6-452d-8084-08924dafa9d7'                                                                                             |
| id                      | 16015b54-dd09-425c-8982-36d440dd224a                                                                                                                                  |
| location                | cloud='', project.domain_id=, project.domain_name='Default', project.id='fc02d48e173c411b9bb6dff1fd65265a', project.name='shiftstack', region_name='regionOne', zone= |
| mac_address             | fa:16:3e:fa:a5:24                                                                                                                                                     |
| name                    |                                                                                                                                                                       |
| network_id              | 79151d07-fe16-44c2-8e9c-3467d79191ba                                                                                                                                  |
| port_security_enabled   | False                                                                                                                                                                 |
| project_id              | fc02d48e173c411b9bb6dff1fd65265a                                                                                                                                      |
| propagate_uplink_status | None                                                                                                                                                                  |
| qos_policy_id           | None                                                                                                                                                                  |
| resource_request        | None                                                                                                                                                                  |
| revision_number         | 3                                                                                                                                                                     |
| security_group_ids      |                                                                                                                                                                       |
| status                  | ACTIVE                                                                                                                                                                |
| tags                    |                                                                                                                                                                       |
| trunk_details           | None                                                                                                                                                                  |
| updated_at              | 2022-03-21T12:29:09Z                                                                                                                                                  |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+

The resources involved are:

$ openstack subnet show restricted_subnet
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                                                                 |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| allocation_pools  | 172.16.0.4-172.16.0.254                                                                                                                                               |
| cidr              | 172.16.0.0/24                                                                                                                                                         |
| created_at        | 2022-03-21T11:57:08Z                                                                                                                                                  |
| description       |                                                                                                                                                                       |
| dns_nameservers   |                                                                                                                                                                       |
| enable_dhcp       | True                                                                                                                                                                  |
| gateway_ip        | 172.16.0.1                                                                                                                                                            |
| host_routes       |                                                                                                                                                                       |
| id                | 878a7964-2ae6-452d-8084-08924dafa9d7                                                                                                                                  |
| ip_version        | 4                                                                                                                                                                     |
| ipv6_address_mode | None                                                                                                                                                                  |
| ipv6_ra_mode      | None                                                                                                                                                                  |
| location          | cloud='', project.domain_id=, project.domain_name='Default', project.id='fc02d48e173c411b9bb6dff1fd65265a', project.name='shiftstack', region_name='regionOne', zone= |
| name              | restricted_subnet                                                                                                                                                     |
| network_id        | 79151d07-fe16-44c2-8e9c-3467d79191ba                                                                                                                                  |
| prefix_length     | None                                                                                                                                                                  |
| project_id        | fc02d48e173c411b9bb6dff1fd65265a                                                                                                                                      |
| revision_number   | 0                                                                                                                                                                     |
| segment_id        | None                                                                                                                                                                  |
| service_types     |                                                                                                                                                                       |
| subnetpool_id     | None                                                                                                                                                                  |
| tags              |                                                                                                                                                                       |
| updated_at        | 2022-03-21T11:57:08Z                                                                                                                                                  |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+

(shiftstack) [cloud-user@installer-host ~]$ openstack network show restricted_network
+---------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                     | Value                                                                                                                                                                 |
+---------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up            | UP                                                                                                                                                                    |
| availability_zone_hints   |                                                                                                                                                                       |
| availability_zones        |                                                                                                                                                                       |
| created_at                | 2022-03-21T11:57:04Z                                                                                                                                                  |
| description               |                                                                                                                                                                       |
| dns_domain                |                                                                                                                                                                       |
| id                        | 79151d07-fe16-44c2-8e9c-3467d79191ba                                                                                                                                  |
| ipv4_address_scope        | None                                                                                                                                                                  |
| ipv6_address_scope        | None                                                                                                                                                                  |
| is_default                | None                                                                                                                                                                  |
| is_vlan_transparent       | None                                                                                                                                                                  |
| location                  | cloud='', project.domain_id=, project.domain_name='Default', project.id='fc02d48e173c411b9bb6dff1fd65265a', project.name='shiftstack', region_name='regionOne', zone= |
| mtu                       | 1442                                                                                                                                                                  |
| name                      | restricted_network                                                                                                                                                    |
| port_security_enabled     | True                                                                                                                                                                  |
| project_id                | fc02d48e173c411b9bb6dff1fd65265a                                                                                                                                      |
| provider:network_type     | None                                                                                                                                                                  |
| provider:physical_network | None                                                                                                                                                                  |
| provider:segmentation_id  | None                                                                                                                                                                  |
| qos_policy_id             | None                                                                                                                                                                  |
| revision_number           | 2                                                                                                                                                                     |
| router:external           | Internal                                                                                                                                                              |
| segments                  | None                                                                                                                                                                  |
| shared                    | False                                                                                                                                                                 |
| status                    | ACTIVE                                                                                                                                                                |
| subnets                   | 878a7964-2ae6-452d-8084-08924dafa9d7                                                                                                                                  |
| tags                      |                                                                                                                                                                       |
| updated_at                | 2022-03-21T11:57:08Z                                                                                                                                                  |
+---------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+


We hit something similar some months ago (not exactly the same) and the issue was related with the tags https://bugzilla.redhat.com/show_bug.cgi?id=1924701

Exactly the same installation destroy is working for >=4.8.


Version-Release number of selected component (if applicable): 4.7.0-0.nightly-2022-03-18-153015 & RHOS-16.1-RHEL-8-20211126.n.1


How reproducible: Always


Steps to Reproduce:
1. Install OCP with IPI using Kuryr on a restricted network without Router connected to the outside world through a proxy.
2. Destroy it.
3.

Actual results: Destroy is stuck.


Expected results: Destroy works.


Additional info: Adding must-gather and installation/destroy logs.