Description of problem: Since https://bugzilla.redhat.com/show_bug.cgi?id=2041893 fix landed, ipi proxy installation with Kuryr is working, but the destroy cluster is not. We are creating a installer_VM with two nics, one connected to the installer_host-subnet with external access, and one connected to restricted_subnet without external access. Then we are installing a proxy on the installer_host and triggering the IPI openshift installation with Kuryr with BYON using the restricted_subnet. CNO is creating a router, because the router that is connected to the installer_host is not visible for the project and it is not connected to the restricted_subnet. The installation is OK, but the destroy is stuck at this point: ERROR {"NeutronError": {"type": "RouterInUse", "message": "Router 5af47047-3abc-44c6-82b6-d97e0fe5bca2 still has ports", "detail": ""}} DEBUG Exiting deleting openstack routers The stuck resources that are: (shiftstack) [cloud-user@installer-host ~]$ openstack router show ostest-sk7dd-external-router +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2022-03-21T12:29:06Z | | description | | | external_gateway_info | null | | flavor_id | None | | id | 5af47047-3abc-44c6-82b6-d97e0fe5bca2 | | interfaces_info | [{"port_id": "16015b54-dd09-425c-8982-36d440dd224a", "ip_address": "172.16.0.1", "subnet_id": "878a7964-2ae6-452d-8084-08924dafa9d7"}] | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='fc02d48e173c411b9bb6dff1fd65265a', project.name='shiftstack', region_name='regionOne', zone= | | name | ostest-sk7dd-external-router | | project_id | fc02d48e173c411b9bb6dff1fd65265a | | revision_number | 160 | | routes | | | status | ACTIVE | | tags | openshiftClusterID=ostest-sk7dd | | updated_at | 2022-03-21T16:27:01Z | +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (shiftstack) [cloud-user@installer-host ~]$ openstack port show 16015b54-dd09-425c-8982-36d440dd224a +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | None | | binding_profile | None | | binding_vif_details | None | | binding_vif_type | None | | binding_vnic_type | normal | | created_at | 2022-03-21T12:29:07Z | | data_plane_status | None | | description | | | device_id | 5af47047-3abc-44c6-82b6-d97e0fe5bca2 | | device_owner | network:router_interface | | dns_assignment | fqdn='host-172-16-0-1.shiftstack.com.', hostname='host-172-16-0-1', ip_address='172.16.0.1' | | dns_domain | | | dns_name | | | extra_dhcp_opts | | | fixed_ips | ip_address='172.16.0.1', subnet_id='878a7964-2ae6-452d-8084-08924dafa9d7' | | id | 16015b54-dd09-425c-8982-36d440dd224a | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='fc02d48e173c411b9bb6dff1fd65265a', project.name='shiftstack', region_name='regionOne', zone= | | mac_address | fa:16:3e:fa:a5:24 | | name | | | network_id | 79151d07-fe16-44c2-8e9c-3467d79191ba | | port_security_enabled | False | | project_id | fc02d48e173c411b9bb6dff1fd65265a | | propagate_uplink_status | None | | qos_policy_id | None | | resource_request | None | | revision_number | 3 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2022-03-21T12:29:09Z | +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+ The resources involved are: $ openstack subnet show restricted_subnet +-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | allocation_pools | 172.16.0.4-172.16.0.254 | | cidr | 172.16.0.0/24 | | created_at | 2022-03-21T11:57:08Z | | description | | | dns_nameservers | | | enable_dhcp | True | | gateway_ip | 172.16.0.1 | | host_routes | | | id | 878a7964-2ae6-452d-8084-08924dafa9d7 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='fc02d48e173c411b9bb6dff1fd65265a', project.name='shiftstack', region_name='regionOne', zone= | | name | restricted_subnet | | network_id | 79151d07-fe16-44c2-8e9c-3467d79191ba | | prefix_length | None | | project_id | fc02d48e173c411b9bb6dff1fd65265a | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2022-03-21T11:57:08Z | +-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (shiftstack) [cloud-user@installer-host ~]$ openstack network show restricted_network +---------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +---------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2022-03-21T11:57:04Z | | description | | | dns_domain | | | id | 79151d07-fe16-44c2-8e9c-3467d79191ba | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='fc02d48e173c411b9bb6dff1fd65265a', project.name='shiftstack', region_name='regionOne', zone= | | mtu | 1442 | | name | restricted_network | | port_security_enabled | True | | project_id | fc02d48e173c411b9bb6dff1fd65265a | | provider:network_type | None | | provider:physical_network | None | | provider:segmentation_id | None | | qos_policy_id | None | | revision_number | 2 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | 878a7964-2ae6-452d-8084-08924dafa9d7 | | tags | | | updated_at | 2022-03-21T11:57:08Z | +---------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+ We hit something similar some months ago (not exactly the same) and the issue was related with the tags https://bugzilla.redhat.com/show_bug.cgi?id=1924701 Exactly the same installation destroy is working for >=4.8. Version-Release number of selected component (if applicable): 4.7.0-0.nightly-2022-03-18-153015 & RHOS-16.1-RHEL-8-20211126.n.1 How reproducible: Always Steps to Reproduce: 1. Install OCP with IPI using Kuryr on a restricted network without Router connected to the outside world through a proxy. 2. Destroy it. 3. Actual results: Destroy is stuck. Expected results: Destroy works. Additional info: Adding must-gather and installation/destroy logs.
Verified on 4.7.48 on top of RHOS-16.2-RHEL-8-20220311.n.1 $ openshift-install --version FATA[0000] Error executing openshift-install: unknown flag: --version [cloud-user@installer-host ~]$ openshift-install version openshift-install 4.7.48 built from commit 4b386e9188695a9e9fc11b4ba848756b62fb94e7 release image registry.ci.openshift.org/ocp/release@sha256:b1fab3144a5bd2e6c4ac2dc2df4eca9deaf942ef9c28519991ab81e593174978 Cluster destroy after ipi proxy installation finished successfully: time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Metadata\" from disk" time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Master Ignition Customization Check\" from disk" time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Worker Ignition Customization Check\" from disk" time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Terraform Variables\" from disk" time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Kubeconfig Admin Client\" from disk" time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Kubeadmin Password\" from disk" time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Certificate (journal-gatewayd)\" from disk" time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Cluster\" from disk" time="2022-04-07T10:58:20-04:00" level=info msg="Time elapsed: 14m12s" $ openstack router list $
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.7.47 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:1166