Bug 1566384

Summary: Ensure kuryr leftovers are deleted upon stack deletion/update
Product: OpenShift Container Platform Reporter: Luis Tomas Bolivar <ltomasbo>
Component: InstallerAssignee: Luis Tomas Bolivar <ltomasbo>
Status: CLOSED CURRENTRELEASE QA Contact: Jon Uriarte <juriarte>
Severity: unspecified Docs Contact:
Priority: high    
Version: unspecifiedCC: aos-bugs, asegurap, jokerman, mmccomas
Target Milestone: ---Keywords: Triaged
Target Release: 3.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-12-21 15:16:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Luis Tomas Bolivar 2018-04-12 08:24:35 UTC
Currently, when deploying OpenShift with kuryr the heat stack creates the required OpenStack resources. However, when the OpenShift installation is later used, Kuryr will create extra OpenStack resources, such as Octavia LoadBalancers or neutron subports.

When deleting the stack (or update it to have a lower number of VM-nodes) some of those resources may not be deleted, which lead the stack deletion/update to fail.

Comment 1 Scott Dodson 2018-08-14 21:24:41 UTC
Should be in openshift-ansible-3.11.0-0.15.0

Comment 2 N. Harrison Ripps 2018-09-21 20:16:28 UTC
Per OCP program call on 21-SEP-2018 we are deferring Kuryr-related bugs to 3.11.z

Comment 3 Jon Uriarte 2018-10-03 16:06:00 UTC
Verified in openshift-ansible-3.11.18-1.git.0.033596c.el7.noarch, on top of OSP 13 2018-09-28.1 puddle.

All the resources, those created by the heat templates and those created by Kuryr, are now deleted with the uninstall playbook.

Verification steps:
1. Install OSP 13 with octavia enabled
2. Install OCP 3.11 with kuryr enabled:
$ ansible-playbook --user openshift -i "/usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py" -i inventory "/usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/prerequisites.yml"
$ ansible-playbook --user openshift -i "/usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py" -i inventory "/usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/provision.yml"
$ ansible-playbook --user openshift -i "/usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py" -i inventory "red-hat-ca.yml"
$ ansible-playbook --user openshift -i "/usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py" -i inventory "repos.yml"
$ ansible-playbook --user openshift -i "/usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py" -i inventory "/usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/install.yml"

3. Create a project, deploymentconfig, scale, and create a service (LB). This step is not strictly necessary as the OCP installation itself includes the deployment of  Load Balancers by Kuryr.
4. Check created Load Balancers and trunk/subports.

$ openstack loadbalancer list                                                                                                                                            
+--------------------------------------+------------------------------------------------+----------------------------------+----------------+---------------------+----------+
| id                                   | name                                           | project_id                       | vip_address    | provisioning_status | provider |
+--------------------------------------+------------------------------------------------+----------------------------------+----------------+---------------------+----------+
| d5ed4ba6-72dc-4969-80d4-93ea3c872a94 | openshift-ansible-openshift.example.com-api-lb | 0127ae37e08b4ef9972632e71f190b14 | 172.30.0.1     | ACTIVE              | octavia  |
| 37564b30-bbb8-43e1-8448-b8a4d350d80e | default/router                                 | 0127ae37e08b4ef9972632e71f190b14 | 172.30.92.240  | ACTIVE              | octavia  |
| bfd8712c-235e-4399-96cd-3b9658a273b2 | default/docker-registry                        | 0127ae37e08b4ef9972632e71f190b14 | 172.30.31.71   | ACTIVE              | octavia  |
| 77d03927-80de-4551-9e1a-b5a3d516f11d | default/registry-console                       | 0127ae37e08b4ef9972632e71f190b14 | 172.30.78.56   | ACTIVE              | octavia  |
| 75314fc7-5f27-4d56-a712-cf1410fc2ee8 | openshift-console/console                      | 0127ae37e08b4ef9972632e71f190b14 | 172.30.23.155  | ACTIVE              | octavia  |
| 3bfd67e2-cb60-405b-b73b-efd9879ef641 | openshift-monitoring/grafana                   | 0127ae37e08b4ef9972632e71f190b14 | 172.30.120.78  | ACTIVE              | octavia  |
| 492e4f62-a80f-4c48-8b9e-eb6a07bccd86 | openshift-monitoring/prometheus-k8s            | 0127ae37e08b4ef9972632e71f190b14 | 172.30.100.165 | ACTIVE              | octavia  |
| 782be4bc-c259-4f0f-8b30-9a05b5918929 | openshift-monitoring/alertmanager-main         | 0127ae37e08b4ef9972632e71f190b14 | 172.30.96.197  | ACTIVE              | octavia  |
| 57873173-c54b-443b-8486-fbbc4cea5fad | test/demo                                      | 0127ae37e08b4ef9972632e71f190b14 | 172.30.227.2   | ACTIVE              | octavia  |
+--------------------------------------+------------------------------------------------+----------------------------------+----------------+---------------------+----------+

$ openstack network trunk list
    +--------------------------------------+------------------------------------+--------------------------------------+-------------+
    | ID                                   | Name                               | Parent Port                          | Description |
    +--------------------------------------+------------------------------------+--------------------------------------+-------------+
    | 3fc651ca-9e6a-4654-a2df-b34cdd13aaf6 | master-0.openshift.example.com     | e0864694-fbe0-4b11-a039-43e44c4314ca |             |
    | 831a0c2c-bbe8-4a94-88ce-dc3cebd5e93b | infra-node-0.openshift.example.com | 3ee3aeaa-9a2d-4436-b0f5-4e043d12473b |             |
    | e3d1e430-59f3-4e78-98a9-708a93501c07 | app-node-0.openshift.example.com   | 25bc3b13-89bc-4e97-8783-d755244001f1 |             |
    | f46e6b48-9530-458e-bbf2-16bd5845fb62 | app-node-1.openshift.example.com   | fdfffed2-8270-4134-a5f3-f3b7f0205af2 |             |
    +--------------------------------------+------------------------------------+--------------------------------------+-------------+
     
     
$ openstack network trunk show app-node-1.openshift.example.com
    +-----------------+--------------------------------------------------------------------------------------------------+
    | Field           | Value                                                                                            |
    +-----------------+--------------------------------------------------------------------------------------------------+
    | admin_state_up  | UP                                                                                               |
    | created_at      | 2018-10-03T10:53:37Z                                                                             |
    | description     |                                                                                                  |
    | id              | f46e6b48-9530-458e-bbf2-16bd5845fb62                                                             |
    | name            | app-node-1.openshift.example.com                                                                 |
    | port_id         | fdfffed2-8270-4134-a5f3-f3b7f0205af2                                                             |
    | project_id      | 0127ae37e08b4ef9972632e71f190b14                                                                 |
    | revision_number | 4                                                                                                |
    | status          | ACTIVE                                                                                           |
    | sub_ports       | port_id='85e5e316-5263-485f-94ef-124d2660a496', segmentation_id='766', segmentation_type='vlan'  |
    |                 | port_id='54568fd3-bdfc-41b4-bb74-99de8b2a1f94', segmentation_id='889', segmentation_type='vlan'  |
    |                 | port_id='643ea964-f19d-4bf3-8815-64f99fe8d361', segmentation_id='2099', segmentation_type='vlan' |
    |                 | port_id='1515173f-22fd-4d4f-956b-e9573be3482a', segmentation_id='2609', segmentation_type='vlan' |
    |                 | port_id='eb6a7401-5d0f-4211-b73b-b02188139ac4', segmentation_id='2822', segmentation_type='vlan' |
    | tags            | []                                                                                               |
    | tenant_id       | 0127ae37e08b4ef9972632e71f190b14                                                                 |
    | updated_at      | 2018-10-03T12:49:35Z                                                                             |
    +-----------------+--------------------------------------------------------------------------------------------------+
     
     
$ openstack network trunk show app-node-0.openshift.example.com                                                                                                          
    +-----------------+--------------------------------------------------------------------------------------------------+
    | Field           | Value                                                                                            |
    +-----------------+--------------------------------------------------------------------------------------------------+
    | admin_state_up  | UP                                                                                               |
    | created_at      | 2018-10-03T10:53:35Z                                                                             |
    | description     |                                                                                                  |
    | id              | e3d1e430-59f3-4e78-98a9-708a93501c07                                                             |
    | name            | app-node-0.openshift.example.com                                                                 |
    | port_id         | 25bc3b13-89bc-4e97-8783-d755244001f1                                                             |
    | project_id      | 0127ae37e08b4ef9972632e71f190b14                                                                 |
    | revision_number | 4                                                                                                |
    | status          | ACTIVE                                                                                           |
    | sub_ports       | port_id='e52f01c0-2faf-4698-9356-f349038f56ea', segmentation_id='258', segmentation_type='vlan'  |
    |                 | port_id='ab85f03f-d738-4080-a960-41e50c7b2927', segmentation_id='1053', segmentation_type='vlan' |
    |                 | port_id='4bc34255-c3a9-4704-b701-b9b94466a953', segmentation_id='1875', segmentation_type='vlan' |
    |                 | port_id='b7c4cebc-6d50-4953-8cfb-41d251483637', segmentation_id='2106', segmentation_type='vlan' |
    |                 | port_id='85766dca-d3ad-477e-b689-b1455aa86ade', segmentation_id='2196', segmentation_type='vlan' |
    | tags            | []                                                                                               |
    | tenant_id       | 0127ae37e08b4ef9972632e71f190b14                                                                 |
    | updated_at      | 2018-10-03T12:45:50Z                                                                             |
    +-----------------+--------------------------------------------------------------------------------------------------+

Check the number of ports in pod subnet:

$ openstack port list | grep 10.11. | wc -l
31

5. Uninstall OCP:
ansible-playbook --user openshift -i "/usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py" -i inventory "/usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/uninstall.yml"

6. Check all the resources have been deleted, including the extra load balancers and subports created by Kuryr:
$ openstack stack list

$ openstack server list
+--------------------------------------+-----------------+--------+--------------------------------------------+----------+----------+
| ID                                   | Name            | Status | Networks                                   | Image    | Flavor   |
+--------------------------------------+-----------------+--------+--------------------------------------------+----------+----------+
| 9a6fafd7-c59e-43b6-99d6-84bd97b8a446 | ansible_host-0  | ACTIVE | private_openshift=172.16.40.7, 10.46.22.41 | rhel-7.5 | m1.small |
| 0e862377-f367-42e6-82a4-d47b4469b9b5 | openshift_dns-0 | ACTIVE | openshift_dns=192.168.23.4, 10.46.22.46    | rhel-7.5 | m1.small |
+--------------------------------------+-----------------+--------+--------------------------------------------+----------+----------+

There are no openshift nodes.

$ openstack loadbalancer list                                                                                                                                            
(empty)
$ openstack port list | grep 10.11. | wc -l                                                                                                                              
0
$ openstack network trunk list
(empty)

Comment 5 Luke Meyer 2018-12-21 15:16:29 UTC
Closing bugs that were verified and targeted for GA but for some reason were not picked up by errata. This bug fix should be present in current 3.11 release content.