Red Hat Bugzilla – Bug 1572131
Service deletion fails when deleting the load balancer
Last modified: 2018-06-27 09:55:07 EDT
Description of problem: Kuryr kubernetes fails when trying to delete the load balancer. The loadbalancer remains in OpenStack although the listener and members are correctly removed. Version-Release number of selected component (if applicable): docker-registry.engineering.redhat.com/rhosp13/openstack-kuryr-controller:latest How reproducible: Always Steps to Reproduce (with OpenShift): 1.- oc new-project test 2.- oc run --image kuryr/demo demo 3.- oc expose dc/demo --port 80 --target-port 8080 4.- wait until octavia LBaaS is up and running 5.- oc delete project test Actual results: 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 274, in on_deleted 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging obj_lbaas.LBaaSServiceSpec()) 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 305, in _sync_lbaas_members 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging if self._sync_lbaas_pools(endpoints, lbaas_state, lbaas_spec): 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 415, in _sync_lbaas_pools 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging if self._sync_lbaas_listeners(endpoints, lbaas_state, lbaas_spec): 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 472, in _sync_lbaas_listeners 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging if self._sync_lbaas_loadbalancer(endpoints, lbaas_state, lbaas_spec): 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 550, in _sync_lbaas_loadbalancer 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging loadbalancer=lb) 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/lbaasv2.py", line 71, in release_loadbalancer 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging lbaas.delete_loadbalancer, loadbalancer.id) 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/lbaasv2.py", line 388, in _release 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging delete(*args, **kwargs) 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1143, in delete_loadbalancer 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging (lbaas_loadbalancer)) 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 350, in delete 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging headers=headers, params=params) 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging headers=headers, params=params) 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging self._handle_fault_response(status_code, replybody, resp) 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging exception_handler_v20(status_code, error_body) 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging request_ids=request_ids) 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging BadRequest: 400-{u'debuginfo': None, u'faultcode': u'Client', u'faultstring': u'Validation failure: Cannot delete Load Balancer 31b6750f-a481-4f4e-b3b8-289edee89a4f - it has children'} 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging Neutron server returns request_ids: ['req-3f6be686-2466-452f-9467-41ff3b65e47f'] 2018-04-26 08:37:06.314 1 ERROR kuryr_kubernetes.handlers.logging Expected results: No error and loadbalancer correctly removed.
Upstream bug: https://bugs.launchpad.net/bugs/1767074
Seems to be fixed on OSP 13, puddle 2018-04-10.2. (tester) [stack@undercloud-0 ~]$ openstack loadbalancer delete LB Validation failure: Cannot delete Load Balancer a1256223-6e9c-4b27-9e4c-a5580029c3b2 - it has children (HTTP 400) (Request-ID: req-8864c5ce-82b2-4afd-aaea-7bbccb01b927) (tester) [stack@undercloud-0 ~]$ openstack loadbalancer delete LB -h usage: openstack loadbalancer delete [-h] [--cascade] <load_balancer> Delete a load balancer positional arguments: <load_balancer> Load balancers to delete (name or ID) optional arguments: -h, --help show this help message and exit --cascade Cascade the delete to all child elements of the load balancer. This command is provided by the python-octaviaclient plugin. (tester) [stack@undercloud-0 ~]$ openstack loadbalancer delete LB --cascade (PASSED)
Looks like this patch comes in with upstream tag 0.4.3 on stable/queens branch
Verified in version openstack-kuryr-kubernetes-controller-0.4.3-1.el7ost.noarch from puddle 20180502.1. 1.- oc new-project test 2.- oc run --image kuryr/demo demo 3.- oc expose dc/demo --port 80 --target-port 8080 4.- Wait until octavia LBaaS is up and running (overcloud) [root@undercloud stack]# openstack loadbalancer list +--------------------------------------+------------------------------------------------+----------------------------------+---------------+---------------------+----------+ | id | name | project_id | vip_address | provisioning_status | provider | +--------------------------------------+------------------------------------------------+----------------------------------+---------------+---------------------+----------+ | a5f0e380-7f17-4665-baa7-b291601742ff | openshift-ansible-openshift.example.com-api-lb | 6c07532860e641989bacc5583275080a | 172.30.0.1 | ACTIVE | octavia | | 23439f39-5b2a-4c4b-9fb1-451fddedde86 | test/demo | 6c07532860e641989bacc5583275080a | 172.30.37.254 | ACTIVE | octavia | +--------------------------------------+------------------------------------------------+----------------------------------+---------------+---------------------+----------+ 5.- oc delete project test 6.- Checked the test/demo load balancer is removed (overcloud) [root@undercloud stack]# openstack loadbalancer list +--------------------------------------+------------------------------------------------+----------------------------------+---------------+---------------------+----------+ | id | name | project_id | vip_address | provisioning_status | provider | +--------------------------------------+------------------------------------------------+----------------------------------+---------------+---------------------+----------+ | a5f0e380-7f17-4665-baa7-b291601742ff | openshift-ansible-openshift.example.com-api-lb | 6c07532860e641989bacc5583275080a | 172.30.0.1 | ACTIVE | octavia | +--------------------------------------+------------------------------------------------+----------------------------------+---------------+---------------------+----------+ And no errors are shown in kuryr log.
(In reply to Noam Manos from comment #2) > Seems to be fixed on OSP 13, puddle 2018-04-10.2. > > > (tester) [stack@undercloud-0 ~]$ openstack loadbalancer delete LB > Validation failure: Cannot delete Load Balancer > a1256223-6e9c-4b27-9e4c-a5580029c3b2 - it has children (HTTP 400) > (Request-ID: req-8864c5ce-82b2-4afd-aaea-7bbccb01b927) > > > (tester) [stack@undercloud-0 ~]$ openstack loadbalancer delete LB -h > usage: openstack loadbalancer delete [-h] [--cascade] <load_balancer> > > Delete a load balancer > > positional arguments: > <load_balancer> Load balancers to delete (name or ID) > > optional arguments: > -h, --help show this help message and exit > --cascade Cascade the delete to all child elements of the load > balancer. > > This command is provided by the python-octaviaclient plugin. > > (tester) [stack@undercloud-0 ~]$ openstack loadbalancer delete LB --cascade > > (PASSED) Thanks for the info Noam!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2086