Bug 1777413 - Race between namespace deletion and member addition
Summary: Race between namespace deletion and member addition
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.4.0
Assignee: Luis Tomas Bolivar
QA Contact: GenadiC
URL:
Whiteboard:
Depends On:
Blocks: 1777424
TreeView+ depends on / blocked
 
Reported: 2019-11-27 15:23 UTC by Luis Tomas Bolivar
Modified: 2020-05-04 11:18 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1777424 (view as bug list)
Environment:
Last Closed: 2020-05-04 11:17:52 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 96 0 'None' closed Bug 1777413: Avoid race between member addition and namespace deletion 2020-04-27 08:44:24 UTC
Launchpad 1853604 0 None None None 2019-11-27 15:23:28 UTC
OpenStack gerrit 695722 0 'None' MERGED Avoid race between member addition and namespace deletion 2020-04-27 08:44:25 UTC
Red Hat Product Errata RHBA-2020:0581 0 None None None 2020-05-04 11:18:16 UTC

Description Luis Tomas Bolivar 2019-11-27 15:23:29 UTC 
When using L2 mode, we need to obtain the member subnet before adding the member to the loadbalancer. If at the same time, on the k8s side, the namespace has been deleted, the next error happens:

2019-11-22 12:21:59.448 1 WARNING kuryr_kubernetes.controller.drivers.namespace_subnet [-] Namespace e2e-kubectl-4495 not found: K8sResourceNotFound: Resource not found: u'{"kind":"Status","apiVersion"[823/1543]│
adata":{},"status":"Failure","message":"namespaces \\"e2e-kubectl-4495\\" not found","reason":"NotFound","details":{"name":"e2e-kubectl-4495","kind":"namespaces"},"code":404}\n' │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler: K8sResourceNotFound: Resource not found: u'{"kind":"Status","apiVersion":"v1","metadata":{},"stat│
us":"Failure","message":"namespaces \\"e2e-kubectl-4495\\" not found","reason":"NotFound","details":{"name":"e2e-kubectl-4495","kind":"namespaces"},"code":404}\n' │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry Traceback (most recent call last): │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__ │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry self._handler(event) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 72, in __call__ │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry self.on_present(obj) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 183, in on_present │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry if self._sync_lbaas_members(endpoints, lbaas_state, lbaas_spec): │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 274, in _sync_lbaas_members │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry self._add_new_members(endpoints, lbaas_state, lbaas_spec)): │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 364, in _add_new_members │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry target_ip) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 400, in _get_pod_subnet │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry subnets_map = self._drv_pod_subnets.get_subnets(pod, project_id) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 47, in get_subnets │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry return self.get_namespace_subnet(pod_namespace) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 51, in get_namespace_subnet │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry subnet_id = self._get_namespace_subnet_id(namespace) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 58, in _get_namespace_subnet_id │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry namespace)) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/k8s_client.py", line 83, in get │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry raise exc.K8sResourceNotFound(response.text) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry K8sResourceNotFound: Resource not found: u'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \\"e2e-kubectl│
-4495\\" not found","reason":"NotFound","details":{"name":"e2e-kubectl-4495","kind":"namespaces"},"code":404}\n' │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry
Comment 2 Itzik Brown 2020-01-27 13:52:00 UTC 
Checking with 4.4.0-0.nightly-2020-01-24-141203
During conformance tests there were 3 restarts but it seems no errors as above.
Comment 4 errata-xmlrpc 2020-05-04 11:17:52 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581
Note You need to log in before you can comment on or make changes to this bug.