Bug 1572131 - Service deletion fails when deleting the load balancer
Summary: Service deletion fails when deleting the load balancer
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-kuryr-kubernetes
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: beta
: 13.0 (Queens)
Assignee: Yossi Boaron
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-26 09:07 UTC by Jon Uriarte
Modified: 2018-06-27 13:55 UTC (History)
3 users (show)

Fixed In Version: openstack-kuryr-kubernetes-0.4.3-1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-27 13:53:50 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 564504 0 None master: MERGED kuryr-kubernetes: Fix octavia lb deleting_race with cascading (I97303b54b900bc40f791309bd2589e617b61177a) 2018-05-02 12:44:30 UTC
OpenStack gerrit 564801 0 None stable/queens: MERGED kuryr-kubernetes: Fix octavia lb deleting_race with cascading (I97303b54b900bc40f791309bd2589e617b61177a) 2018-05-02 12:44:24 UTC
Red Hat Product Errata RHEA-2018:2086 0 None None None 2018-06-27 13:55:07 UTC

Description Jon Uriarte 2018-04-26 09:07:37 UTC
Description of problem:
Kuryr kubernetes fails when trying to delete the load balancer.
The loadbalancer remains in OpenStack although the listener and members are correctly removed.

Version-Release number of selected component (if applicable):
docker-registry.engineering.redhat.com/rhosp13/openstack-kuryr-controller:latest

How reproducible: Always

Steps to Reproduce (with OpenShift):
 1.- oc new-project test
 2.- oc run --image kuryr/demo demo
 3.- oc expose dc/demo --port 80 --target-port 8080
 4.- wait until octavia LBaaS is up and running
 5.- oc delete project test

Actual results:
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 274, in on_deleted
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging     obj_lbaas.LBaaSServiceSpec())
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 305, in _sync_lbaas_members
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging     if self._sync_lbaas_pools(endpoints, lbaas_state, lbaas_spec):
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 415, in _sync_lbaas_pools
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging     if self._sync_lbaas_listeners(endpoints, lbaas_state, lbaas_spec):
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 472, in _sync_lbaas_listeners
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging     if self._sync_lbaas_loadbalancer(endpoints, lbaas_state, lbaas_spec):
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 550, in _sync_lbaas_loadbalancer
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging     loadbalancer=lb)
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/lbaasv2.py", line 71, in release_loadbalancer
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging     lbaas.delete_loadbalancer, loadbalancer.id)
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/lbaasv2.py", line 388, in _release
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging     delete(*args, **kwargs)
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1143, in delete_loadbalancer
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging     (lbaas_loadbalancer))
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 350, in delete
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging     headers=headers, params=params)
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging     headers=headers, params=params)
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging     self._handle_fault_response(status_code, replybody, resp)
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging     exception_handler_v20(status_code, error_body)
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging     request_ids=request_ids)
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging BadRequest: 400-{u'debuginfo': None, u'faultcode': u'Client', u'faultstring': u'Validation failure: Cannot delete Load Balancer 31b6750f-a481-4f4e-b3b8-289edee89a4f - it has children'}
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging Neutron server returns request_ids: ['req-3f6be686-2466-452f-9467-41ff3b65e47f']
2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging

Expected results:
No error and loadbalancer correctly removed.

Comment 1 Jon Uriarte 2018-04-26 09:10:15 UTC
Upstream bug: https://bugs.launchpad.net/bugs/1767074

Comment 2 Noam Manos 2018-04-30 10:39:38 UTC
Seems to be fixed on OSP 13, puddle 2018-04-10.2.


(tester) [stack@undercloud-0 ~]$ openstack loadbalancer delete LB
Validation failure: Cannot delete Load Balancer a1256223-6e9c-4b27-9e4c-a5580029c3b2 - it has children (HTTP 400) (Request-ID: req-8864c5ce-82b2-4afd-aaea-7bbccb01b927)


(tester) [stack@undercloud-0 ~]$ openstack loadbalancer delete LB -h
usage: openstack loadbalancer delete [-h] [--cascade] <load_balancer>

Delete a load balancer

positional arguments:
  <load_balancer>  Load balancers to delete (name or ID)

optional arguments:
  -h, --help       show this help message and exit
  --cascade        Cascade the delete to all child elements of the load
                   balancer.

This command is provided by the python-octaviaclient plugin.

(tester) [stack@undercloud-0 ~]$ openstack loadbalancer delete LB --cascade

      (PASSED)

Comment 6 Jon Schlueter 2018-05-02 12:46:25 UTC
Looks like this patch comes in with upstream tag 0.4.3 on stable/queens branch

Comment 11 Jon Uriarte 2018-05-08 08:14:33 UTC
Verified in version openstack-kuryr-kubernetes-controller-0.4.3-1.el7ost.noarch from puddle 20180502.1.

1.- oc new-project test
2.- oc run --image kuryr/demo demo
3.- oc expose dc/demo --port 80 --target-port 8080
4.- Wait until octavia LBaaS is up and running

 (overcloud) [root@undercloud stack]# openstack loadbalancer list
 +--------------------------------------+------------------------------------------------+----------------------------------+---------------+---------------------+----------+
 | id                                   | name                                           | project_id                       | vip_address   | provisioning_status | provider |
 +--------------------------------------+------------------------------------------------+----------------------------------+---------------+---------------------+----------+
 | a5f0e380-7f17-4665-baa7-b291601742ff | openshift-ansible-openshift.example.com-api-lb | 6c07532860e641989bacc5583275080a | 172.30.0.1    | ACTIVE              | octavia  |
 | 23439f39-5b2a-4c4b-9fb1-451fddedde86 | test/demo                                      | 6c07532860e641989bacc5583275080a | 172.30.37.254 | ACTIVE              | octavia  |
 +--------------------------------------+------------------------------------------------+----------------------------------+---------------+---------------------+----------+

5.- oc delete project test
6.- Checked the test/demo load balancer is removed
 (overcloud) [root@undercloud stack]# openstack loadbalancer list
 +--------------------------------------+------------------------------------------------+----------------------------------+---------------+---------------------+----------+
 | id                                   | name                                           | project_id                       | vip_address   | provisioning_status | provider |
 +--------------------------------------+------------------------------------------------+----------------------------------+---------------+---------------------+----------+
 | a5f0e380-7f17-4665-baa7-b291601742ff | openshift-ansible-openshift.example.com-api-lb | 6c07532860e641989bacc5583275080a | 172.30.0.1    | ACTIVE              | octavia  |
 +--------------------------------------+------------------------------------------------+----------------------------------+---------------+---------------------+----------+

And no errors are shown in kuryr log.

Comment 12 Jon Uriarte 2018-05-08 08:15:10 UTC
(In reply to Noam Manos from comment #2)
> Seems to be fixed on OSP 13, puddle 2018-04-10.2.
> 
> 
> (tester) [stack@undercloud-0 ~]$ openstack loadbalancer delete LB
> Validation failure: Cannot delete Load Balancer
> a1256223-6e9c-4b27-9e4c-a5580029c3b2 - it has children (HTTP 400)
> (Request-ID: req-8864c5ce-82b2-4afd-aaea-7bbccb01b927)
> 
> 
> (tester) [stack@undercloud-0 ~]$ openstack loadbalancer delete LB -h
> usage: openstack loadbalancer delete [-h] [--cascade] <load_balancer>
> 
> Delete a load balancer
> 
> positional arguments:
>   <load_balancer>  Load balancers to delete (name or ID)
> 
> optional arguments:
>   -h, --help       show this help message and exit
>   --cascade        Cascade the delete to all child elements of the load
>                    balancer.
> 
> This command is provided by the python-octaviaclient plugin.
> 
> (tester) [stack@undercloud-0 ~]$ openstack loadbalancer delete LB --cascade
> 
>       (PASSED)

Thanks for the info Noam!

Comment 14 errata-xmlrpc 2018-06-27 13:53:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.