Bug 1777413
| Summary: | Race between namespace deletion and member addition | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Luis Tomas Bolivar <ltomasbo> | |
| Component: | Networking | Assignee: | Luis Tomas Bolivar <ltomasbo> | |
| Networking sub component: | kuryr | QA Contact: | GenadiC <gcheresh> | |
| Status: | CLOSED ERRATA | Docs Contact: | ||
| Severity: | medium | |||
| Priority: | unspecified | CC: | itbrown | |
| Version: | 4.4 | |||
| Target Milestone: | --- | |||
| Target Release: | 4.4.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | No Doc Update | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1777424 (view as bug list) | Environment: | ||
| Last Closed: | 2020-05-04 11:17:52 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1777424 | |||
Checking with 4.4.0-0.nightly-2020-01-24-141203 During conformance tests there were 3 restarts but it seems no errors as above. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581 |
When using L2 mode, we need to obtain the member subnet before adding the member to the loadbalancer. If at the same time, on the k8s side, the namespace has been deleted, the next error happens: 2019-11-22 12:21:59.448 1 WARNING kuryr_kubernetes.controller.drivers.namespace_subnet [-] Namespace e2e-kubectl-4495 not found: K8sResourceNotFound: Resource not found: u'{"kind":"Status","apiVersion"[823/1543]│ adata":{},"status":"Failure","message":"namespaces \\"e2e-kubectl-4495\\" not found","reason":"NotFound","details":{"name":"e2e-kubectl-4495","kind":"namespaces"},"code":404}\n' │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler: K8sResourceNotFound: Resource not found: u'{"kind":"Status","apiVersion":"v1","metadata":{},"stat│ us":"Failure","message":"namespaces \\"e2e-kubectl-4495\\" not found","reason":"NotFound","details":{"name":"e2e-kubectl-4495","kind":"namespaces"},"code":404}\n' │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry Traceback (most recent call last): │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__ │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry self._handler(event) │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 72, in __call__ │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry self.on_present(obj) │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 183, in on_present │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry if self._sync_lbaas_members(endpoints, lbaas_state, lbaas_spec): │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 274, in _sync_lbaas_members │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry self._add_new_members(endpoints, lbaas_state, lbaas_spec)): │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 364, in _add_new_members │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry target_ip) │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 400, in _get_pod_subnet │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry subnets_map = self._drv_pod_subnets.get_subnets(pod, project_id) │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 47, in get_subnets │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry return self.get_namespace_subnet(pod_namespace) │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 51, in get_namespace_subnet │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry subnet_id = self._get_namespace_subnet_id(namespace) │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 58, in _get_namespace_subnet_id │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry namespace)) │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/k8s_client.py", line 83, in get │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry raise exc.K8sResourceNotFound(response.text) │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry K8sResourceNotFound: Resource not found: u'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \\"e2e-kubectl│ -4495\\" not found","reason":"NotFound","details":{"name":"e2e-kubectl-4495","kind":"namespaces"},"code":404}\n' │ 2019-11-22 12:21:59.448 1 ERROR kuryr_kubernetes.handlers.retry