Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1777424

Summary: Race between namespace deletion and member addition
Product: OpenShift Container Platform Reporter: Luis Tomas Bolivar <ltomasbo>
Component: NetworkingAssignee: Luis Tomas Bolivar <ltomasbo>
Networking sub component: kuryr QA Contact: GenadiC <gcheresh>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: gcheresh, itbrown
Version: 4.4   
Target Milestone: ---   
Target Release: 4.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1777413 Environment:
Last Closed: 2020-01-23 11:14:45 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1777413    
Bug Blocks:    

Description Luis Tomas Bolivar 2019-11-27 15:25:47 UTC 
+++ This bug was initially created as a clone of Bug #1777413 +++

When using L2 mode, we need to obtain the member subnet before adding the member to the loadbalancer. If at the same time, on the k8s side, the namespace has been deleted, the next error happens:

2019-11-22 12:21:59.448 1 WARNING kuryr_kubernetes.controller.drivers.namespace_subnet [-] Namespace e2e-kubectl-4495 not found: K8sResourceNotFound: Resource not found: u'{"kind":"Status","apiVersion"[823/1543]│
adata":{},"status":"Failure","message":"namespaces \\"e2e-kubectl-4495\\" not found","reason":"NotFound","details":{"name":"e2e-kubectl-4495","kind":"namespaces"},"code":404}\n' │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler: K8sResourceNotFound: Resource not found: u'{"kind":"Status","apiVersion":"v1","metadata":{},"stat│
us":"Failure","message":"namespaces \\"e2e-kubectl-4495\\" not found","reason":"NotFound","details":{"name":"e2e-kubectl-4495","kind":"namespaces"},"code":404}\n' │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry Traceback (most recent call last): │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__ │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry self._handler(event) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 72, in __call__ │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry self.on_present(obj) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 183, in on_present │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry if self._sync_lbaas_members(endpoints, lbaas_state, lbaas_spec): │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 274, in _sync_lbaas_members │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry self._add_new_members(endpoints, lbaas_state, lbaas_spec)): │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 364, in _add_new_members │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry target_ip) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 400, in _get_pod_subnet │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry subnets_map = self._drv_pod_subnets.get_subnets(pod, project_id) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 47, in get_subnets │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry return self.get_namespace_subnet(pod_namespace) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 51, in get_namespace_subnet │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry subnet_id = self._get_namespace_subnet_id(namespace) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 58, in _get_namespace_subnet_id │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry namespace)) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/k8s_client.py", line 83, in get │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry raise exc.K8sResourceNotFound(response.text) │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry K8sResourceNotFound: Resource not found: u'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \\"e2e-kubectl│
-4495\\" not found","reason":"NotFound","details":{"name":"e2e-kubectl-4495","kind":"namespaces"},"code":404}\n' │
2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry
Comment 2 Itzik Brown 2019-12-11 14:35:09 UTC 
Ran conformance tests.                                                             
The following failed:                                                              
[k8s.io] Kubelet when scheduling a busybox command that always fails in a pod should have an terminated reason [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-cli] Kubectl client [k8s.io] Kubectl logs should be able to retrieve and filter logs  [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-network] DNS should provide /etc/hosts entries for the cluster [LinuxOnly] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-network] DNS should provide DNS for services  [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-network] DNS should provide DNS for the cluster  [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-storage] ConfigMap should be consumable from pods in volume [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-storage] Projected downwardAPI should provide container's memory limit [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
                                                                                   
No restart of the kuryr-controller                                                 
                                                                                   
Using 4.3.0-0.nightly-2019-12-11-073659                                                                                                                                                                            
~
Comment 4 errata-xmlrpc 2020-01-23 11:14:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062