Bug 1993364

How to reproduce it:

* Deploy OCP (e.g. 4.9), with networkType set to Kuryr
* In install-config, use a pre-provisionned network & router (using machinesSubnet)

After a successful deployment, run the destroy command.
The command will finish but won't actually remove the subnets from the pre provisionned router, and therefore won't remove the networks.

This leaves a lot of unused resources on the OpenStack cloud.

FYI I'm using https://github.com/openshift/release/pull/21092 to reproduce it in CI.

Verified in 4.10.0-0.nightly-2021-09-15-031230 on top of OSP 16.1.6 (RHOS-16.1-RHEL-8-20210818.n.0).

# Create a network, a subnet and a router
openstack network create net1
openstack subnet create subnet1 --network net1 --subnet-range 172.18.0.0/24 --allocation-pool start=172.18.0.10,end=172.18.0.254 --dns-nameserver 10.46.0.31
openstack router create router1
openstack router add subnet router1 subnet1
openstack router set router1 --external-gateway nova

# Create a port in net1 to be attached to the installer_host
openstack port create --network net1 --security-group installer_host-sg --host <installer_host ID>

# Update the install-config:

networking:
  clusterNetworks:
  - cidr:             10.128.0.0/14
    hostSubnetLength: 9
  serviceCIDR: 172.30.0.0/16
  machineCIDR: 172.18.0.0/24
  type: "Kuryr"|"OpenShiftSDN"
platform:
  openstack:
    cloud:            "shiftstack"
    externalNetwork:  ""
    region:           "regionOne"
    computeFlavor:    "m4.xlarge"
    machinesSubnet: d2257ad6-b68a-4a6c-ae55-7eefe5d7e03d
    apiVIP: "172.18.0.5"
    ingressVIP: "172.18.0.7"


# Update the /etc/hosts:

172.18.0.5 api.ostest.shiftstack.com
172.18.0.7 oauth-openshift.apps.ostest.shiftstack.com
172.18.0.7 console-openshift-console.apps.ostest.shiftstack.com
172.18.0.7 downloads-openshift-console.apps.ostest.shiftstack.com
172.18.0.7 canary-openshift-ingress-canary.apps.ostest.shiftstack.com
172.18.0.7 alertmanager-main-openshift-monitoring.apps.ostest.shiftstack.com
172.18.0.7 grafana-openshift-monitoring.apps.ostest.shiftstack.com
172.18.0.7 prometheus-k8s-openshift-monitoring.apps.ostest.shiftstack.com
172.18.0.7 thanos-querier-openshift-monitoring.apps.ostest.shiftstack.com


Run the installer with Kuryr:
$ openshift-install create cluster --dir=ostest

INFO Install complete!

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2021-09-15-031230   True        False         5m47s   Cluster version is 4.10.0-0.nightly-2021-09-15-031230


Create a LB type svc:
oc new-project test1-ns
oc create deployment test1-dep --image=quay.io/kuryr/demo
oc scale deployments/test1-dep --replicas=2
oc expose deployment test1-dep --name test1-svc --type=LoadBalancer --port 80 --target-port=8080

Check the svc creation in OCP (it needs to take a fip from OSP):
$ oc get svc -o wide
NAME        TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)        AGE     SELECTOR
test1-svc   LoadBalancer   172.30.116.172   10.46.22.249   80:32512/TCP   4m18s   app=test1-dep

Check the LB and fip creation in OSP:
$ openstack loadbalancer list | grep test
| 6888f5bb-eb07-407f-8f77-cd729a039511 | test1-ns/test1-svc                                                           | 5b27b448f1964691905a79b758eb4f6a | 172.30.116.172 | ACTIVE              | ovn      |

$ openstack floating ip list
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| ID                                   | Floating IP Address | Fixed IP Address | Port                                 | Floating Network                     | Project                          |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
...
| e92a1528-950d-4144-a863-688f57956d45 | 10.46.22.249        | 172.30.116.172   | bf3aa82b-6d9a-41df-ad93-a356f42e9b26 | b1b1d60d-ebda-4999-a8fd-673fe8a83083 | 5b27b448f1964691905a79b758eb4f6a |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+


Destroy the cluster:
$ openshift-install destroy cluster --dir=ostest

INFO Time elapsed: 8m23s

Check all the installer-created subnets were removed:
$ openstack subnet list
+--------------------------------------+-----------------------+--------------------------------------+----------------+
| ID                                   | Name                  | Network                              | Subnet         |
+--------------------------------------+-----------------------+--------------------------------------+----------------+
| 1234e81f-bbb0-4888-bb24-8c066f5a69f1 | installer_host-subnet | 3316ac0d-34d7-48b2-b461-d4efa88a0c6b | 172.16.40.0/24 |
| b28e0dad-2094-43bd-b5be-de900299c306 | restricted_subnet     | c299ca00-b7da-46a4-b23a-cf98fc8f5f47 | 172.16.0.0/24  |
| d2257ad6-b68a-4a6c-ae55-7eefe5d7e03d | subnet1               | 162aa054-a2e2-4600-a47e-5c737d3493b5 | 172.18.0.0/24  |
+--------------------------------------+-----------------------+--------------------------------------+----------------+

The above subnets are the pre-created ones so it's ok.

Check the fip for the LB type svc has been removed as well.

Removing the Triaged keyword because:

* the QE automation assessment (flag qe_test_coverage) is missing

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056