Bug 1566384 - Ensure kuryr leftovers are deleted upon stack deletion/update
Summary: Ensure kuryr leftovers are deleted upon stack deletion/update
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: unspecified
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: ---
: 3.11.0
Assignee: Luis Tomas Bolivar
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-12 08:24 UTC by Luis Tomas Bolivar
Modified: 2018-12-21 15:16 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-12-21 15:16:29 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift openshift-ansible pull 8254 0 None None None 2018-05-03 15:26:17 UTC

Description Luis Tomas Bolivar 2018-04-12 08:24:35 UTC
Currently, when deploying OpenShift with kuryr the heat stack creates the required OpenStack resources. However, when the OpenShift installation is later used, Kuryr will create extra OpenStack resources, such as Octavia LoadBalancers or neutron subports.

When deleting the stack (or update it to have a lower number of VM-nodes) some of those resources may not be deleted, which lead the stack deletion/update to fail.

Comment 1 Scott Dodson 2018-08-14 21:24:41 UTC
Should be in openshift-ansible-3.11.0-0.15.0

Comment 2 N. Harrison Ripps 2018-09-21 20:16:28 UTC
Per OCP program call on 21-SEP-2018 we are deferring Kuryr-related bugs to 3.11.z

Comment 3 Jon Uriarte 2018-10-03 16:06:00 UTC
Verified in openshift-ansible-3.11.18-1.git.0.033596c.el7.noarch, on top of OSP 13 2018-09-28.1 puddle.

All the resources, those created by the heat templates and those created by Kuryr, are now deleted with the uninstall playbook.

Verification steps:
1. Install OSP 13 with octavia enabled
2. Install OCP 3.11 with kuryr enabled:
$ ansible-playbook --user openshift -i "/usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py" -i inventory "/usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/prerequisites.yml"
$ ansible-playbook --user openshift -i "/usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py" -i inventory "/usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/provision.yml"
$ ansible-playbook --user openshift -i "/usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py" -i inventory "red-hat-ca.yml"
$ ansible-playbook --user openshift -i "/usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py" -i inventory "repos.yml"
$ ansible-playbook --user openshift -i "/usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py" -i inventory "/usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/install.yml"

3. Create a project, deploymentconfig, scale, and create a service (LB). This step is not strictly necessary as the OCP installation itself includes the deployment of  Load Balancers by Kuryr.
4. Check created Load Balancers and trunk/subports.

$ openstack loadbalancer list                                                                                                                                            
+--------------------------------------+------------------------------------------------+----------------------------------+----------------+---------------------+----------+
| id                                   | name                                           | project_id                       | vip_address    | provisioning_status | provider |
+--------------------------------------+------------------------------------------------+----------------------------------+----------------+---------------------+----------+
| d5ed4ba6-72dc-4969-80d4-93ea3c872a94 | openshift-ansible-openshift.example.com-api-lb | 0127ae37e08b4ef9972632e71f190b14 | 172.30.0.1     | ACTIVE              | octavia  |
| 37564b30-bbb8-43e1-8448-b8a4d350d80e | default/router                                 | 0127ae37e08b4ef9972632e71f190b14 | 172.30.92.240  | ACTIVE              | octavia  |
| bfd8712c-235e-4399-96cd-3b9658a273b2 | default/docker-registry                        | 0127ae37e08b4ef9972632e71f190b14 | 172.30.31.71   | ACTIVE              | octavia  |
| 77d03927-80de-4551-9e1a-b5a3d516f11d | default/registry-console                       | 0127ae37e08b4ef9972632e71f190b14 | 172.30.78.56   | ACTIVE              | octavia  |
| 75314fc7-5f27-4d56-a712-cf1410fc2ee8 | openshift-console/console                      | 0127ae37e08b4ef9972632e71f190b14 | 172.30.23.155  | ACTIVE              | octavia  |
| 3bfd67e2-cb60-405b-b73b-efd9879ef641 | openshift-monitoring/grafana                   | 0127ae37e08b4ef9972632e71f190b14 | 172.30.120.78  | ACTIVE              | octavia  |
| 492e4f62-a80f-4c48-8b9e-eb6a07bccd86 | openshift-monitoring/prometheus-k8s            | 0127ae37e08b4ef9972632e71f190b14 | 172.30.100.165 | ACTIVE              | octavia  |
| 782be4bc-c259-4f0f-8b30-9a05b5918929 | openshift-monitoring/alertmanager-main         | 0127ae37e08b4ef9972632e71f190b14 | 172.30.96.197  | ACTIVE              | octavia  |
| 57873173-c54b-443b-8486-fbbc4cea5fad | test/demo                                      | 0127ae37e08b4ef9972632e71f190b14 | 172.30.227.2   | ACTIVE              | octavia  |
+--------------------------------------+------------------------------------------------+----------------------------------+----------------+---------------------+----------+

$ openstack network trunk list
    +--------------------------------------+------------------------------------+--------------------------------------+-------------+
    | ID                                   | Name                               | Parent Port                          | Description |
    +--------------------------------------+------------------------------------+--------------------------------------+-------------+
    | 3fc651ca-9e6a-4654-a2df-b34cdd13aaf6 | master-0.openshift.example.com     | e0864694-fbe0-4b11-a039-43e44c4314ca |             |
    | 831a0c2c-bbe8-4a94-88ce-dc3cebd5e93b | infra-node-0.openshift.example.com | 3ee3aeaa-9a2d-4436-b0f5-4e043d12473b |             |
    | e3d1e430-59f3-4e78-98a9-708a93501c07 | app-node-0.openshift.example.com   | 25bc3b13-89bc-4e97-8783-d755244001f1 |             |
    | f46e6b48-9530-458e-bbf2-16bd5845fb62 | app-node-1.openshift.example.com   | fdfffed2-8270-4134-a5f3-f3b7f0205af2 |             |
    +--------------------------------------+------------------------------------+--------------------------------------+-------------+
     
     
$ openstack network trunk show app-node-1.openshift.example.com
    +-----------------+--------------------------------------------------------------------------------------------------+
    | Field           | Value                                                                                            |
    +-----------------+--------------------------------------------------------------------------------------------------+
    | admin_state_up  | UP                                                                                               |
    | created_at      | 2018-10-03T10:53:37Z                                                                             |
    | description     |                                                                                                  |
    | id              | f46e6b48-9530-458e-bbf2-16bd5845fb62                                                             |
    | name            | app-node-1.openshift.example.com                                                                 |
    | port_id         | fdfffed2-8270-4134-a5f3-f3b7f0205af2                                                             |
    | project_id      | 0127ae37e08b4ef9972632e71f190b14                                                                 |
    | revision_number | 4                                                                                                |
    | status          | ACTIVE                                                                                           |
    | sub_ports       | port_id='85e5e316-5263-485f-94ef-124d2660a496', segmentation_id='766', segmentation_type='vlan'  |
    |                 | port_id='54568fd3-bdfc-41b4-bb74-99de8b2a1f94', segmentation_id='889', segmentation_type='vlan'  |
    |                 | port_id='643ea964-f19d-4bf3-8815-64f99fe8d361', segmentation_id='2099', segmentation_type='vlan' |
    |                 | port_id='1515173f-22fd-4d4f-956b-e9573be3482a', segmentation_id='2609', segmentation_type='vlan' |
    |                 | port_id='eb6a7401-5d0f-4211-b73b-b02188139ac4', segmentation_id='2822', segmentation_type='vlan' |
    | tags            | []                                                                                               |
    | tenant_id       | 0127ae37e08b4ef9972632e71f190b14                                                                 |
    | updated_at      | 2018-10-03T12:49:35Z                                                                             |
    +-----------------+--------------------------------------------------------------------------------------------------+
     
     
$ openstack network trunk show app-node-0.openshift.example.com                                                                                                          
    +-----------------+--------------------------------------------------------------------------------------------------+
    | Field           | Value                                                                                            |
    +-----------------+--------------------------------------------------------------------------------------------------+
    | admin_state_up  | UP                                                                                               |
    | created_at      | 2018-10-03T10:53:35Z                                                                             |
    | description     |                                                                                                  |
    | id              | e3d1e430-59f3-4e78-98a9-708a93501c07                                                             |
    | name            | app-node-0.openshift.example.com                                                                 |
    | port_id         | 25bc3b13-89bc-4e97-8783-d755244001f1                                                             |
    | project_id      | 0127ae37e08b4ef9972632e71f190b14                                                                 |
    | revision_number | 4                                                                                                |
    | status          | ACTIVE                                                                                           |
    | sub_ports       | port_id='e52f01c0-2faf-4698-9356-f349038f56ea', segmentation_id='258', segmentation_type='vlan'  |
    |                 | port_id='ab85f03f-d738-4080-a960-41e50c7b2927', segmentation_id='1053', segmentation_type='vlan' |
    |                 | port_id='4bc34255-c3a9-4704-b701-b9b94466a953', segmentation_id='1875', segmentation_type='vlan' |
    |                 | port_id='b7c4cebc-6d50-4953-8cfb-41d251483637', segmentation_id='2106', segmentation_type='vlan' |
    |                 | port_id='85766dca-d3ad-477e-b689-b1455aa86ade', segmentation_id='2196', segmentation_type='vlan' |
    | tags            | []                                                                                               |
    | tenant_id       | 0127ae37e08b4ef9972632e71f190b14                                                                 |
    | updated_at      | 2018-10-03T12:45:50Z                                                                             |
    +-----------------+--------------------------------------------------------------------------------------------------+

Check the number of ports in pod subnet:

$ openstack port list | grep 10.11. | wc -l
31

5. Uninstall OCP:
ansible-playbook --user openshift -i "/usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py" -i inventory "/usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/uninstall.yml"

6. Check all the resources have been deleted, including the extra load balancers and subports created by Kuryr:
$ openstack stack list

$ openstack server list
+--------------------------------------+-----------------+--------+--------------------------------------------+----------+----------+
| ID                                   | Name            | Status | Networks                                   | Image    | Flavor   |
+--------------------------------------+-----------------+--------+--------------------------------------------+----------+----------+
| 9a6fafd7-c59e-43b6-99d6-84bd97b8a446 | ansible_host-0  | ACTIVE | private_openshift=172.16.40.7, 10.46.22.41 | rhel-7.5 | m1.small |
| 0e862377-f367-42e6-82a4-d47b4469b9b5 | openshift_dns-0 | ACTIVE | openshift_dns=192.168.23.4, 10.46.22.46    | rhel-7.5 | m1.small |
+--------------------------------------+-----------------+--------+--------------------------------------------+----------+----------+

There are no openshift nodes.

$ openstack loadbalancer list                                                                                                                                            
(empty)
$ openstack port list | grep 10.11. | wc -l                                                                                                                              
0
$ openstack network trunk list
(empty)

Comment 5 Luke Meyer 2018-12-21 15:16:29 UTC
Closing bugs that were verified and targeted for GA but for some reason were not picked up by errata. This bug fix should be present in current 3.11 release content.


Note You need to log in before you can comment on or make changes to this bug.