Bug 2066406 - Cluster destroy fails when using byo with Kuryr and there is no pre-created router
Summary: Cluster destroy fails when using byo with Kuryr and there is no pre-created r...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.7
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.7.z
Assignee: Michał Dulko
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On: 1940159
Blocks: 2080219
TreeView+ depends on / blocked
 
Reported: 2022-03-21 16:55 UTC by rlobillo
Modified: 2022-04-29 09:42 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2080219 (view as bug list)
Environment:
Last Closed: 2022-04-11 11:28:50 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 5754 0 None open Bug 2066406: OpenStack+Kuryr: Fix `cluster destroy` with BYON 2022-03-28 14:54:57 UTC
Red Hat Product Errata RHSA-2022:1166 0 None None None 2022-04-11 11:29:00 UTC

Description rlobillo 2022-03-21 16:55:09 UTC
Description of problem:

Since https://bugzilla.redhat.com/show_bug.cgi?id=2041893 fix landed, ipi proxy installation with Kuryr is working, but the destroy cluster is not.

We are creating a installer_VM with two nics, one connected to the installer_host-subnet with external access, and one connected to restricted_subnet without external access.
Then we are installing a proxy on the installer_host and triggering the IPI openshift installation with Kuryr with BYON using the restricted_subnet. CNO is creating a router, because the router that is connected to the installer_host is not visible for the project and it is not connected to the restricted_subnet.

The installation is OK, but the destroy is stuck at this point:

ERROR {"NeutronError": {"type": "RouterInUse", "message": "Router 5af47047-3abc-44c6-82b6-d97e0fe5bca2 still has ports", "detail": ""}}                                                      
DEBUG Exiting deleting openstack routers

The stuck resources that are:

(shiftstack) [cloud-user@installer-host ~]$ openstack router show ostest-sk7dd-external-router
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+                                          
| Field                   | Value                                                                                                                                                                 |                                          
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+                                          
| admin_state_up          | UP                                                                                                                                                                    |                                          
| availability_zone_hints |                                                                                                                                                                       |                                          
| availability_zones      |                                                                                                                                                                       |                                          
| created_at              | 2022-03-21T12:29:06Z                                                                                                                                                  |                                          
| description             |                                                                                                                                                                       |                                          
| external_gateway_info   | null                                                                                                                                                                  |                                          
| flavor_id               | None                                                                                                                                                                  |                                          
| id                      | 5af47047-3abc-44c6-82b6-d97e0fe5bca2                                                                                                                                  |                                          
| interfaces_info         | [{"port_id": "16015b54-dd09-425c-8982-36d440dd224a", "ip_address": "172.16.0.1", "subnet_id": "878a7964-2ae6-452d-8084-08924dafa9d7"}]                                |                                          
| location                | cloud='', project.domain_id=, project.domain_name='Default', project.id='fc02d48e173c411b9bb6dff1fd65265a', project.name='shiftstack', region_name='regionOne', zone= |                                          
| name                    | ostest-sk7dd-external-router                                                                                                                                          |                                          
| project_id              | fc02d48e173c411b9bb6dff1fd65265a                                                                                                                                      |                                          
| revision_number         | 160                                                                                                                                                                   |                                          
| routes                  |                                                                                                                                                                       |                                          
| status                  | ACTIVE                                                                                                                                                                |                                          
| tags                    | openshiftClusterID=ostest-sk7dd                                                                                                                                       |                                          
| updated_at              | 2022-03-21T16:27:01Z                                                                                                                                                  |                                          
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+                                    

(shiftstack) [cloud-user@installer-host ~]$ openstack port show 16015b54-dd09-425c-8982-36d440dd224a
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                                                 |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | UP                                                                                                                                                                    |
| allowed_address_pairs   |                                                                                                                                                                       |
| binding_host_id         | None                                                                                                                                                                  |
| binding_profile         | None                                                                                                                                                                  |
| binding_vif_details     | None                                                                                                                                                                  |
| binding_vif_type        | None                                                                                                                                                                  |
| binding_vnic_type       | normal                                                                                                                                                                |
| created_at              | 2022-03-21T12:29:07Z                                                                                                                                                  |
| data_plane_status       | None                                                                                                                                                                  |
| description             |                                                                                                                                                                       |
| device_id               | 5af47047-3abc-44c6-82b6-d97e0fe5bca2                                                                                                                                  |
| device_owner            | network:router_interface                                                                                                                                              |
| dns_assignment          | fqdn='host-172-16-0-1.shiftstack.com.', hostname='host-172-16-0-1', ip_address='172.16.0.1'                                                                           |
| dns_domain              |                                                                                                                                                                       |
| dns_name                |                                                                                                                                                                       |
| extra_dhcp_opts         |                                                                                                                                                                       |
| fixed_ips               | ip_address='172.16.0.1', subnet_id='878a7964-2ae6-452d-8084-08924dafa9d7'                                                                                             |
| id                      | 16015b54-dd09-425c-8982-36d440dd224a                                                                                                                                  |
| location                | cloud='', project.domain_id=, project.domain_name='Default', project.id='fc02d48e173c411b9bb6dff1fd65265a', project.name='shiftstack', region_name='regionOne', zone= |
| mac_address             | fa:16:3e:fa:a5:24                                                                                                                                                     |
| name                    |                                                                                                                                                                       |
| network_id              | 79151d07-fe16-44c2-8e9c-3467d79191ba                                                                                                                                  |
| port_security_enabled   | False                                                                                                                                                                 |
| project_id              | fc02d48e173c411b9bb6dff1fd65265a                                                                                                                                      |
| propagate_uplink_status | None                                                                                                                                                                  |
| qos_policy_id           | None                                                                                                                                                                  |
| resource_request        | None                                                                                                                                                                  |
| revision_number         | 3                                                                                                                                                                     |
| security_group_ids      |                                                                                                                                                                       |
| status                  | ACTIVE                                                                                                                                                                |
| tags                    |                                                                                                                                                                       |
| trunk_details           | None                                                                                                                                                                  |
| updated_at              | 2022-03-21T12:29:09Z                                                                                                                                                  |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+

The resources involved are:

$ openstack subnet show restricted_subnet
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                                                                 |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| allocation_pools  | 172.16.0.4-172.16.0.254                                                                                                                                               |
| cidr              | 172.16.0.0/24                                                                                                                                                         |
| created_at        | 2022-03-21T11:57:08Z                                                                                                                                                  |
| description       |                                                                                                                                                                       |
| dns_nameservers   |                                                                                                                                                                       |
| enable_dhcp       | True                                                                                                                                                                  |
| gateway_ip        | 172.16.0.1                                                                                                                                                            |
| host_routes       |                                                                                                                                                                       |
| id                | 878a7964-2ae6-452d-8084-08924dafa9d7                                                                                                                                  |
| ip_version        | 4                                                                                                                                                                     |
| ipv6_address_mode | None                                                                                                                                                                  |
| ipv6_ra_mode      | None                                                                                                                                                                  |
| location          | cloud='', project.domain_id=, project.domain_name='Default', project.id='fc02d48e173c411b9bb6dff1fd65265a', project.name='shiftstack', region_name='regionOne', zone= |
| name              | restricted_subnet                                                                                                                                                     |
| network_id        | 79151d07-fe16-44c2-8e9c-3467d79191ba                                                                                                                                  |
| prefix_length     | None                                                                                                                                                                  |
| project_id        | fc02d48e173c411b9bb6dff1fd65265a                                                                                                                                      |
| revision_number   | 0                                                                                                                                                                     |
| segment_id        | None                                                                                                                                                                  |
| service_types     |                                                                                                                                                                       |
| subnetpool_id     | None                                                                                                                                                                  |
| tags              |                                                                                                                                                                       |
| updated_at        | 2022-03-21T11:57:08Z                                                                                                                                                  |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+

(shiftstack) [cloud-user@installer-host ~]$ openstack network show restricted_network
+---------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                     | Value                                                                                                                                                                 |
+---------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up            | UP                                                                                                                                                                    |
| availability_zone_hints   |                                                                                                                                                                       |
| availability_zones        |                                                                                                                                                                       |
| created_at                | 2022-03-21T11:57:04Z                                                                                                                                                  |
| description               |                                                                                                                                                                       |
| dns_domain                |                                                                                                                                                                       |
| id                        | 79151d07-fe16-44c2-8e9c-3467d79191ba                                                                                                                                  |
| ipv4_address_scope        | None                                                                                                                                                                  |
| ipv6_address_scope        | None                                                                                                                                                                  |
| is_default                | None                                                                                                                                                                  |
| is_vlan_transparent       | None                                                                                                                                                                  |
| location                  | cloud='', project.domain_id=, project.domain_name='Default', project.id='fc02d48e173c411b9bb6dff1fd65265a', project.name='shiftstack', region_name='regionOne', zone= |
| mtu                       | 1442                                                                                                                                                                  |
| name                      | restricted_network                                                                                                                                                    |
| port_security_enabled     | True                                                                                                                                                                  |
| project_id                | fc02d48e173c411b9bb6dff1fd65265a                                                                                                                                      |
| provider:network_type     | None                                                                                                                                                                  |
| provider:physical_network | None                                                                                                                                                                  |
| provider:segmentation_id  | None                                                                                                                                                                  |
| qos_policy_id             | None                                                                                                                                                                  |
| revision_number           | 2                                                                                                                                                                     |
| router:external           | Internal                                                                                                                                                              |
| segments                  | None                                                                                                                                                                  |
| shared                    | False                                                                                                                                                                 |
| status                    | ACTIVE                                                                                                                                                                |
| subnets                   | 878a7964-2ae6-452d-8084-08924dafa9d7                                                                                                                                  |
| tags                      |                                                                                                                                                                       |
| updated_at                | 2022-03-21T11:57:08Z                                                                                                                                                  |
+---------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+


We hit something similar some months ago (not exactly the same) and the issue was related with the tags https://bugzilla.redhat.com/show_bug.cgi?id=1924701

Exactly the same installation destroy is working for >=4.8.


Version-Release number of selected component (if applicable): 4.7.0-0.nightly-2022-03-18-153015 & RHOS-16.1-RHEL-8-20211126.n.1


How reproducible: Always


Steps to Reproduce:
1. Install OCP with IPI using Kuryr on a restricted network without Router connected to the outside world through a proxy.
2. Destroy it.
3.

Actual results: Destroy is stuck.


Expected results: Destroy works.


Additional info: Adding must-gather and installation/destroy logs.

Comment 5 rlobillo 2022-04-07 14:59:20 UTC
Verified on 4.7.48 on top of RHOS-16.2-RHEL-8-20220311.n.1

$ openshift-install --version
FATA[0000] Error executing openshift-install: unknown flag: --version 
[cloud-user@installer-host ~]$ openshift-install version
openshift-install 4.7.48
built from commit 4b386e9188695a9e9fc11b4ba848756b62fb94e7
release image registry.ci.openshift.org/ocp/release@sha256:b1fab3144a5bd2e6c4ac2dc2df4eca9deaf942ef9c28519991ab81e593174978


Cluster destroy after ipi proxy installation finished successfully:

time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Metadata\" from disk"
time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Master Ignition Customization Check\" from disk"
time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Worker Ignition Customization Check\" from disk"
time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Terraform Variables\" from disk"
time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Kubeconfig Admin Client\" from disk"
time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Kubeadmin Password\" from disk"
time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Certificate (journal-gatewayd)\" from disk"
time="2022-04-07T10:58:20-04:00" level=debug msg="Purging asset \"Cluster\" from disk"
time="2022-04-07T10:58:20-04:00" level=info msg="Time elapsed: 14m12s"


$ openstack router list

$

Comment 7 errata-xmlrpc 2022-04-11 11:28:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.7.47 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1166


Note You need to log in before you can comment on or make changes to this bug.