Bug 1777424 - Race between namespace deletion and member addition
Summary: Race between namespace deletion and member addition
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.3.0
Assignee: Luis Tomas Bolivar
QA Contact: GenadiC
URL:
Whiteboard:
Depends On: 1777413
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-27 15:25 UTC by Luis Tomas Bolivar
Modified: 2020-01-23 11:14 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1777413
Environment:
Last Closed: 2020-01-23 11:14:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 97 0 'None' closed [release-4.3] Bug 1777424: Avoid race between member addition and namespace deletion 2020-03-25 10:46:18 UTC

Description Luis Tomas Bolivar 2019-11-27 15:25:47 UTC
+++ This bug was initially created as a clone of Bug #1777413 +++

When using L2 mode, we need to obtain the member subnet before adding the member to the loadbalancer. If at the same time, on the k8s side, the namespace has been deleted, the next error happens:

2019-11-22 12:21:59.448 1 WARNING kuryr_kubernetes.controller.drivers.namespace_subnet [-] Namespace e2e-kubectl-4495 not found: K8sResourceNotFound: Resource not found: u'{"kind":"Status","apiVersion"[823/1543]│
adata":{},"status":"Failure","message":"namespaces \\"e2e-kubectl-4495\\" not found","reason":"NotFound","details":{"name":"e2e-kubectl-4495","kind":"namespaces"},"code":404}\n' │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler: K8sResourceNotFound: Resource not found: u'{"kind":"Status","apiVersion":"v1","metadata":{},"stat│
us":"Failure","message":"namespaces \\"e2e-kubectl-4495\\" not found","reason":"NotFound","details":{"name":"e2e-kubectl-4495","kind":"namespaces"},"code":404}\n' │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry Traceback (most recent call last): │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__ │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry self._handler(event) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 72, in __call__ │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry self.on_present(obj) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 183, in on_present │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry if self._sync_lbaas_members(endpoints, lbaas_state, lbaas_spec): │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 274, in _sync_lbaas_members │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry self._add_new_members(endpoints, lbaas_state, lbaas_spec)): │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 364, in _add_new_members │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry target_ip) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 400, in _get_pod_subnet │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry subnets_map = self._drv_pod_subnets.get_subnets(pod, project_id) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 47, in get_subnets │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry return self.get_namespace_subnet(pod_namespace) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 51, in get_namespace_subnet │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry subnet_id = self._get_namespace_subnet_id(namespace) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 58, in _get_namespace_subnet_id │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry namespace)) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/k8s_client.py", line 83, in get │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry raise exc.K8sResourceNotFound(response.text) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry K8sResourceNotFound: Resource not found: u'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \\"e2e-kubectl│
-4495\\" not found","reason":"NotFound","details":{"name":"e2e-kubectl-4495","kind":"namespaces"},"code":404}\n' │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry

Comment 2 Itzik Brown 2019-12-11 14:35:09 UTC
Ran conformance tests.                                                             
The following failed:                                                              
[k8s.io] Kubelet when scheduling a busybox command that always fails in a pod should have an terminated reason [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-cli] Kubectl client [k8s.io] Kubectl logs should be able to retrieve and filter logs  [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-network] DNS should provide /etc/hosts entries for the cluster [LinuxOnly] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-network] DNS should provide DNS for services  [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-network] DNS should provide DNS for the cluster  [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-storage] ConfigMap should be consumable from pods in volume [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-storage] Projected downwardAPI should provide container's memory limit [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
                                                                                   
No restart of the kuryr-controller                                                 
                                                                                   
Using 4.3.0-0.nightly-2019-12-11-073659                                                                                                                                                                            
~

Comment 4 errata-xmlrpc 2020-01-23 11:14:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062


Note You need to log in before you can comment on or make changes to this bug.