Bug 1784440 - [Kuryr]Ensure leftover load balancers are deleted upon Kuryr-Controller start
Summary: [Kuryr]Ensure leftover load balancers are deleted upon Kuryr-Controller start
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.3.0
Assignee: Maysa Macedo
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On: 1784435
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-17 12:51 UTC by Luis Tomas Bolivar
Modified: 2020-01-23 11:19 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1784435
Environment:
Last Closed: 2020-01-23 11:19:28 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 112 0 None closed [release-4.3] bug 1784440: Lbaas deletion 2020-01-27 15:03:02 UTC
Github openshift kuryr-kubernetes pull 133 0 None closed [release-4.3] Bug 1784440: Ensure only lb with no corresponding svc is cleaned up 2020-01-27 15:03:02 UTC
Red Hat Product Errata RHBA-2020:0062 0 None None None 2020-01-23 11:19:54 UTC

Description Luis Tomas Bolivar 2019-12-17 12:51:37 UTC
+++ This bug was initially created as a clone of Bug #1784435 +++

Description of problem:

When the deletion of a service is triggered and the load balancer is still being provisioned, if the Kuryr-controller restarts, the deletion
event will be gone and the load balancer remains on the cluster with no associated service.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Jon Uriarte 2019-12-24 12:49:07 UTC
Failed QA in 4.3.0-0.nightly-2019-12-23-120031 on 2019-12-06.2 OSP 13 puddle.

Verification steps:

1. Create project, pod and service
 $ oc new-project test
 $ oc run --image kuryr/demo pod1
 $ oc expose dc/pod1 --port 80 --target-port 8080

 $ oc get pods
 NAME            READY   STATUS      RESTARTS   AGE
 pod1-1-deploy   0/1     Completed   0          5m46s
 pod1-1-svcsv    1/1     Running     0          5m12s

 $ oc get svc
 NAME   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
 pod1   ClusterIP   172.30.27.29   <none>        80/TCP    4m5s

2. Check the LB is created in Openstack
 $ openstack loadbalancer list
 +--------------------------------------+-----------+----------------------------------+--------------+---------------------+----------+
 | id                                   | name      | project_id                       | vip_address  | provisioning_status | provider |        
 +--------------------------------------+-----------+----------------------------------+--------------+---------------------+----------+
 | ...                                                                                                                                 |
 | c223b194-6106-493e-9c11-a4fc2efba060 | test/pod1 | 5a37f509dae046128af2549545893cdb | 172.30.27.29 | ACTIVE              | octavia  |
 +--------------------------------------+-----------+----------------------------------+--------------+---------------------+----------+

3. Delete the service
 $ oc delete svc pod1
 service "pod1" deleted 

4. Delete kuryr-controller pod during LB deletion
 $ oc -n openshift-kuryr delete pod kuryr-controller-56bb5c995-hwp59
 pod "kuryr-controller-56bb5c995-hwp59" deleted

 $ openstack loadbalancer list
 +--------------------------------------+-------------------------------------+--------------+---------------------+----------+
 | id                                   | name                                | vip_address  | provisioning_status | provider |        
 +--------------------------------------+-------------------------------------+--------------+---------------------+----------+
 | ...                                                                                                                        |
 | e73f3d82-abc8-4f82-a0b6-160d7bb7cddc | ostest-gjpjs-kuryr-api-loadbalancer | 172.30.0.1   | PENDING_DELETE      | octavia  |
 | c223b194-6106-493e-9c11-a4fc2efba060 | test/pod1                           | 172.30.27.29 | PENDING_DELETE      | octavia  |
 +--------------------------------------+-------------------------------------+--------------+---------------------+----------+

5. Check the LB has been deleted in Openstack, even with kuryr-controller pod restart
There is no test/pod1 load balancer in `openstack loadbalancer list` but API LB has been deleted as well when it shouldn't.

Reproducer:
- Deploy OCP
- Delete kuryr controller pod
- API LB should not be recreated

Comment 8 Jon Uriarte 2020-01-16 10:50:10 UTC
Verified in 4.3.0-0.nightly-2020-01-16-031402 on 2019-12-13.1 OSP 13 puddle.

Verification steps:

1. Create project, pod and service
 $ oc new-project test
 $ oc run --image kuryr/demo pod1
 $ oc expose dc/pod1 --port 80 --target-port 8080

 $ oc get pods
 NAME            READY   STATUS      RESTARTS   AGE
 pod1-1-deploy   0/1     Completed   0          2m31s
 pod1-1-ld2p7    1/1     Running     0          2m7s

 $ oc get svc
 NAME   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
 pod1   ClusterIP   172.30.26.178   <none>        80/TCP    5s

2. Check the LB is created in Openstack
 $ openstack loadbalancer list
 +--------------------------------------+-----------+----------------------------------+---------------+---------------------+----------+
 | id                                   | name      | project_id                       | vip_address   | provisioning_status | provider |        
 +--------------------------------------+-----------+----------------------------------+---------------+---------------------+----------+
 | ...                                                                                                                                  |
 | ccf4687e-852b-4f0b-9e38-df2fef8bc4c6 | test/pod1 | a8217d2b58cf45db928a7d421d11042f | 172.30.26.178 | ACTIVE              | octavia  |
 +--------------------------------------+-----------+----------------------------------+---------------+---------------------+----------+

3. Delete the service
 $ oc delete svc pod1
 service "pod1" deleted 

4. Delete kuryr-controller pod during LB deletion
 $ oc -n openshift-kuryr delete pod kuryr-controller-77b669b58b-fvwzw
 pod "kuryr-controller-77b669b58b-fvwzw" deleted

 $ openstack loadbalancer list
 +--------------------------------------+-------------------------------------+---------------+---------------------+----------+
 | id                                   | name                                | vip_address   | provisioning_status | provider |        
 +--------------------------------------+-------------------------------------+---------------+---------------------+----------+
 | ...                                                                                                                         |
 | daf4c997-79b0-4e0c-a60a-8cecd0d47fb4 | ostest-2scsr-kuryr-api-loadbalancer | 172.30.0.1    | ACTIVE              | octavia  |
 | ccf4687e-852b-4f0b-9e38-df2fef8bc4c6 | test/pod1                           | 172.30.26.178 | PENDING_DELETE      | octavia  |
 +--------------------------------------+-------------------------------------+---------------+---------------------+----------+

5. Check the LB has been deleted in Openstack, even with kuryr-controller pod restart
There is no test/pod1 load balancer in `openstack loadbalancer list` and the API LB has not been deleted this time.

Comment 10 errata-xmlrpc 2020-01-23 11:19:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062


Note You need to log in before you can comment on or make changes to this bug.