Bug 1582480 - lbaas kuryr handler dies due to endpoint without pods
Summary: lbaas kuryr handler dies due to endpoint without pods
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-kuryr-kubernetes
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: Upstream M3
: 14.0 (Rocky)
Assignee: Luis Tomas Bolivar
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-25 11:26 UTC by Luis Tomas Bolivar
Modified: 2019-01-11 11:50 UTC (History)
2 users (show)

Fixed In Version: openstack-kuryr-kubernetes-0.4.2-0.20180404104924.985c387.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-11 11:49:59 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1773349 0 None None None 2018-05-25 11:29:38 UTC
OpenStack gerrit 570562 0 None None None 2018-05-25 11:29:16 UTC
OpenStack gerrit 570586 0 None None None 2018-05-25 15:01:33 UTC
Red Hat Product Errata RHEA-2019:0045 0 None None None 2019-01-11 11:50:16 UTC

Description Luis Tomas Bolivar 2018-05-25 11:26:10 UTC
Description of problem:

There seems to be a race when deleting lbaas members if the deployment is being deleted too (for instance when deleting a namespace that contains the deploy and the svc. This leads to try to delete the lbaas members using an endpoint that has no pods, therefore failing with the next error:
2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {u'object': {u'kind': u'Endpoints', u'subsets': None, u'apiVersion': u'v1', u'metadata': {u'name'
: u'demo', u'labels': {u'run': u'demo'}, u'namespace': u'test', u'resourceVersion': u'4685', u'creationTimestamp': u'2018-05-25T07:57:33Z', u'annotations': {u'openstack.org/kuryr-lbaas-spec'
: u'{"versioned_object.data": {"ip": "10.0.0.182", "lb_ip": null, "ports": [{"versioned_object.data": {"name": null, "port": 80, "protocol": "TCP"}, "versioned_object.name": "LBaaSPortSpec",
 "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}], "project_id": "e5073e1e1ef740fb8a0ade68c60876e2", "security_groups_ids": ["58d0a017-303a-4cb1-aa81-b80
e797394ce"], "subnet_id": "b46ad50b-58a3-43e4-99bf-c6eafcf660f2", "type": "ClusterIP"}, "versioned_object.name": "LBaaSServiceSpec", "versioned_object.namespace": "kuryr_kubernetes", "versio
ned_object.version": "1.0"}', u'openstack.org/kuryr-lbaas-state': u'{"versioned_object.data": {"listeners": [{"versioned_object.changes": ["id"], "versioned_object.data": {"id": "11b45277-84
a1-4263-a037-2f98024fa3d5", "loadbalancer_id": "7e8eea49-76f1-4522-a9e1-c0ef0f377fc1", "name": "test/demo:TCP:80", "port": 80, "project_id": "e5073e1e1ef740fb8a0ade68c60876e2", "protocol": "
TCP"}, "versioned_object.name": "LBaaSListener", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}], "loadbalancer": {"versioned_object.data": {"id": "7e8e
ea49-76f1-4522-a9e1-c0ef0f377fc1", "ip": "10.0.0.182", "name": "test/demo", "port_id": "5656e004-fe95-477b-8327-c9d3e10efb39", "project_id": "e5073e1e1ef740fb8a0ade68c60876e2", "provider": "
haproxy", "security_groups": ["58d0a017-303a-4cb1-aa81-b80e797394ce"], "subnet_id": "b46ad50b-58a3-43e4-99bf-c6eafcf660f2"}, "versioned_object.name": "LBaaSLoadBalancer", "versioned_object.n
amespace": "kuryr_kubernetes", "versioned_object.version": "1.1"}, "members": [{"versioned_object.changes": ["id"], "versioned_object.data": {"id": "a0e3e625-6989-44e3-88f8-d4a05269c999", "i
p": "10.0.1.143", "name": "test/demo-5995548848-lk6vz:8080", "pool_id": "14582762-f755-484a-8ff5-8ed657b46935", "port": 8080, "project_id": "e5073e1e1ef740fb8a0ade68c60876e2", "subnet_id": "
b46ad50b-58a3-43e4-99bf-c6eafcf660f2"}, "versioned_object.name": "LBaaSMember", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}, {"versioned_object.chang
es": ["id"], "versioned_object.data": {"id": "0dfb4fc7-8cbb-41ce-8d39-205a0c4229f0", "ip": "10.0.1.132", "name": "test/demo-5995548848-2scc6:8080", "pool_id": "14582762-f755-484a-8ff5-8ed657
b46935", "port": 8080, "project_id": "e5073e1e1ef740fb8a0ade68c60876e2", "subnet_id": "b46ad50b-58a3-43e4-99bf-c6eafcf660f2"}, "versioned_object.name": "LBaaSMember", "versioned_object.names
pace": "kuryr_kubernetes", "versioned_object.version": "1.0"}, {"versioned_object.changes": ["id"], "versioned_object.data": {"id": "1f8f16b0-8e19-4bac-b362-fa3898399823", "ip": "10.0.1.136"
, "name": "test/demo-5995548848-jpsm2:8080", "pool_id": "14582762-f755-484a-8ff5-8ed657b46935", "port": 8080, "project_id": "e5073e1e1ef740fb8a0ade68c60876e2", "subnet_id": "b46ad50b-58a3-43
e4-99bf-c6eafcf660f2"}, "versioned_object.name": "LBaaSMember", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}], "pools": [{"versioned_object.changes":
["id"], "versioned_object.data": {"id": "14582762-f755-484a-8ff5-8ed657b46935", "listener_id": "11b45277-84a1-4263-a037-2f98024fa3d5", "loadbalancer_id": "7e8eea49-76f1-4522-a9e1-c0ef0f377fc
1", "name": "test/demo:TCP:80", "project_id": "e5073e1e1ef740fb8a0ade68c60876e2", "protocol": "TCP"}, "versioned_object.name": "LBaaSPool", "versioned_object.namespace": "kuryr_kubernetes",
"versioned_object.version": "1.0"}], "service_pub_ip_info": null}, "versioned_object.name": "LBaaSState", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}
'}, u'selfLink': u'/api/v1/namespaces/test/endpoints/demo', u'uid': u'484f4dbb-5ff1-11e8-9edc-fa163ef1032e'}}, u'type': u'DELETED'}: TypeError: 'NoneType' object is not iterable
2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__
2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/retry.py", line 55, in __call__
2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 77, in __call__
2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging self.on_deleted(obj)
2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 284, in on_deleted
2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging obj_lbaas.LBaaSServiceSpec())
2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 312, in _sync_lbaas_members
2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging if self._remove_unused_members(endpoints, lbaas_state, lbaas_spec):
2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 402, in _remove_unused_members
2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging for s in endpoints['subsets']
2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging TypeError: 'NoneType' object is not iterable


How reproducible:
100 %

Steps to Reproduce:
1. kubectl create namespace test                                                                          
namespace "test" created
2. kubectl run --image kuryr/demo demo -n test
3. kubectl scale -n test deploy/demo --replicas=3
4. kubectl expose -n test deploy/demo --port 80 --target-port 8080
5. wait until the service is up (curl the IP of the service and receive the response from the pods)
6. kubectl delete namespace test


Additional info:
- This part of the code is only called for LBaaSV2, so Octavia deployments will not be affected
- If the kuryr-controller healthcheck probes are enabled, there kuryr-controller pod will get restarted but the neutron lbaas resources will be left behind

Comment 7 errata-xmlrpc 2019-01-11 11:49:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045


Note You need to log in before you can comment on or make changes to this bug.