Description of problem: There seems to be a race when deleting lbaas members if the deployment is being deleted too (for instance when deleting a namespace that contains the deploy and the svc. This leads to try to delete the lbaas members using an endpoint that has no pods, therefore failing with the next error: 2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {u'object': {u'kind': u'Endpoints', u'subsets': None, u'apiVersion': u'v1', u'metadata': {u'name' : u'demo', u'labels': {u'run': u'demo'}, u'namespace': u'test', u'resourceVersion': u'4685', u'creationTimestamp': u'2018-05-25T07:57:33Z', u'annotations': {u'openstack.org/kuryr-lbaas-spec' : u'{"versioned_object.data": {"ip": "10.0.0.182", "lb_ip": null, "ports": [{"versioned_object.data": {"name": null, "port": 80, "protocol": "TCP"}, "versioned_object.name": "LBaaSPortSpec", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}], "project_id": "e5073e1e1ef740fb8a0ade68c60876e2", "security_groups_ids": ["58d0a017-303a-4cb1-aa81-b80 e797394ce"], "subnet_id": "b46ad50b-58a3-43e4-99bf-c6eafcf660f2", "type": "ClusterIP"}, "versioned_object.name": "LBaaSServiceSpec", "versioned_object.namespace": "kuryr_kubernetes", "versio ned_object.version": "1.0"}', u'openstack.org/kuryr-lbaas-state': u'{"versioned_object.data": {"listeners": [{"versioned_object.changes": ["id"], "versioned_object.data": {"id": "11b45277-84 a1-4263-a037-2f98024fa3d5", "loadbalancer_id": "7e8eea49-76f1-4522-a9e1-c0ef0f377fc1", "name": "test/demo:TCP:80", "port": 80, "project_id": "e5073e1e1ef740fb8a0ade68c60876e2", "protocol": " TCP"}, "versioned_object.name": "LBaaSListener", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}], "loadbalancer": {"versioned_object.data": {"id": "7e8e ea49-76f1-4522-a9e1-c0ef0f377fc1", "ip": "10.0.0.182", "name": "test/demo", "port_id": "5656e004-fe95-477b-8327-c9d3e10efb39", "project_id": "e5073e1e1ef740fb8a0ade68c60876e2", "provider": " haproxy", "security_groups": ["58d0a017-303a-4cb1-aa81-b80e797394ce"], "subnet_id": "b46ad50b-58a3-43e4-99bf-c6eafcf660f2"}, "versioned_object.name": "LBaaSLoadBalancer", "versioned_object.n amespace": "kuryr_kubernetes", "versioned_object.version": "1.1"}, "members": [{"versioned_object.changes": ["id"], "versioned_object.data": {"id": "a0e3e625-6989-44e3-88f8-d4a05269c999", "i p": "10.0.1.143", "name": "test/demo-5995548848-lk6vz:8080", "pool_id": "14582762-f755-484a-8ff5-8ed657b46935", "port": 8080, "project_id": "e5073e1e1ef740fb8a0ade68c60876e2", "subnet_id": " b46ad50b-58a3-43e4-99bf-c6eafcf660f2"}, "versioned_object.name": "LBaaSMember", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}, {"versioned_object.chang es": ["id"], "versioned_object.data": {"id": "0dfb4fc7-8cbb-41ce-8d39-205a0c4229f0", "ip": "10.0.1.132", "name": "test/demo-5995548848-2scc6:8080", "pool_id": "14582762-f755-484a-8ff5-8ed657 b46935", "port": 8080, "project_id": "e5073e1e1ef740fb8a0ade68c60876e2", "subnet_id": "b46ad50b-58a3-43e4-99bf-c6eafcf660f2"}, "versioned_object.name": "LBaaSMember", "versioned_object.names pace": "kuryr_kubernetes", "versioned_object.version": "1.0"}, {"versioned_object.changes": ["id"], "versioned_object.data": {"id": "1f8f16b0-8e19-4bac-b362-fa3898399823", "ip": "10.0.1.136" , "name": "test/demo-5995548848-jpsm2:8080", "pool_id": "14582762-f755-484a-8ff5-8ed657b46935", "port": 8080, "project_id": "e5073e1e1ef740fb8a0ade68c60876e2", "subnet_id": "b46ad50b-58a3-43 e4-99bf-c6eafcf660f2"}, "versioned_object.name": "LBaaSMember", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}], "pools": [{"versioned_object.changes": ["id"], "versioned_object.data": {"id": "14582762-f755-484a-8ff5-8ed657b46935", "listener_id": "11b45277-84a1-4263-a037-2f98024fa3d5", "loadbalancer_id": "7e8eea49-76f1-4522-a9e1-c0ef0f377fc 1", "name": "test/demo:TCP:80", "project_id": "e5073e1e1ef740fb8a0ade68c60876e2", "protocol": "TCP"}, "versioned_object.name": "LBaaSPool", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"}], "service_pub_ip_info": null}, "versioned_object.name": "LBaaSState", "versioned_object.namespace": "kuryr_kubernetes", "versioned_object.version": "1.0"} '}, u'selfLink': u'/api/v1/namespaces/test/endpoints/demo', u'uid': u'484f4dbb-5ff1-11e8-9edc-fa163ef1032e'}}, u'type': u'DELETED'}: TypeError: 'NoneType' object is not iterable 2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last): 2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__ 2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) 2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/retry.py", line 55, in __call__ 2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) 2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 77, in __call__ 2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging self.on_deleted(obj) 2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 284, in on_deleted 2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging obj_lbaas.LBaaSServiceSpec()) 2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 312, in _sync_lbaas_members 2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging if self._remove_unused_members(endpoints, lbaas_state, lbaas_spec): 2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 402, in _remove_unused_members 2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging for s in endpoints['subsets'] 2018-05-25 07:59:20.283 1 ERROR kuryr_kubernetes.handlers.logging TypeError: 'NoneType' object is not iterable How reproducible: 100 % Steps to Reproduce: 1. kubectl create namespace test namespace "test" created 2. kubectl run --image kuryr/demo demo -n test 3. kubectl scale -n test deploy/demo --replicas=3 4. kubectl expose -n test deploy/demo --port 80 --target-port 8080 5. wait until the service is up (curl the IP of the service and receive the response from the pods) 6. kubectl delete namespace test Additional info: - This part of the code is only called for LBaaSV2, so Octavia deployments will not be affected - If the kuryr-controller healthcheck probes are enabled, there kuryr-controller pod will get restarted but the neutron lbaas resources will be left behind
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045