Bug 1677562
| Summary: | Race when deleting namespaces with network policies | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Luis Tomas Bolivar <ltomasbo> |
| Component: | openstack-kuryr-kubernetes | Assignee: | Luis Tomas Bolivar <ltomasbo> |
| Status: | CLOSED ERRATA | QA Contact: | GenadiC <gcheresh> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 15.0 (Stein) | CC: | asegurap |
| Target Milestone: | Upstream M3 | Keywords: | Triaged |
| Target Release: | 15.0 (Stein) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-kuryr-kubernetes-0.6.2-0.20190305141049.a019712.el8ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-09-21 11:20:21 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811 |
When using network policies, there seems to be a race when deleting a namespace that contains pods, services and network policies. When that happens, the next error appears on the kuryr-controller, making it failing the healtchecks and therefore forcing its restart: 2019-02-14 16:22:48.880 1 DEBUG kuryr_kubernetes.k8s_client [-] Delete /apis/openstack.org/v1/namespaces/test/kuryrnetpolicies/np-np delete /usr/lib/python2.7/site-packages/kuryr_kubernetes/k8s_client.py:122 2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy [-] Kubernetes Client Exception deleting kuryrnetpolicy CRD.: K8sClientException: <Response [404]> 2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy Traceback (most recent call last): 2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 468, in _del_kuryrnetpolicy _crd 2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy netpolicy_crd_name)) 2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/k8s_client.py", line 132, in delete 2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy raise exc.K8sClientException(response) 2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy K8sClientException: <Response [404]> 2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry [-] Report handler unhealthy NetworkPolicyHandler: K8sClientException: <Response [404]> 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry Traceback (most recent call last): 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/retry.py", line 56, in __call__ 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry self._handler(event) 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 77, in __call__ 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry self.on_deleted(obj) 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/policy.py", line 114, in on_deleted 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry self._drv_policy.release_network_policy(netpolicy_crd) 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 391, in release_network_policy 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry netpolicy_crd['metadata']['namespace']) 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 468, in _del_kuryrnetpolicy_crd 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry netpolicy_crd_name)) 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/k8s_client.py", line 132, in delete 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry raise exc.K8sClientException(response) 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry K8sClientException: <Response [404]> 2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {u'object': {u'kind': u'NetworkPolicy', u'spec': {u'policyTypes': [u'Ingress'], u'ingress': [{u'from': [{u'podSelector' : {u'matchLabels': {u'run': u'no-demo'}}}], u'ports': [{u'protocol': u'TCP', u'port': 8080}]}], u'podSelector': {u'matchLabels': {u'run': u'demo'}}}, u'apiVersion': u'networking.k8s.io/v1', u'metadata': {u'name': u'np', u'generation': 1, u'namespace': u'test', u'resourceVersion': u'12504', u'creationTimestamp': u'2019-02-14T15:43:38Z', u'annotations': {u'kuryrnetpolicy_selfLink': u'/apis/openstack.org/v1/namespaces/test/ kuryrnetpolicies/np-np', u'kubectl.kubernetes.io/last-applied-configuration': u'{"apiVersion":"networking.k8s.io/v1","kind":"NetworkPolicy","metadata":{"annotations":{},"name":"np","namespace":"test"},"spec":{"in gress":[{"from":[{"podSelector":{"matchLabels":{"run":"no-demo"}}}],"ports":[{"port":8080,"protocol":"TCP"}]}],"podSelector":{"matchLabels":{"run":"demo"}},"policyTypes":["Ingress"]}}\n'}, u'selfLink': u'/apis/ne tworking.k8s.io/v1/namespaces/test/networkpolicies/np', u'uid': u'4c2447ec-306f-11e9-b3e0-842b2b2d5107'}}, u'type': u'DELETED'}: K8sClientException: <Response [404]> 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last): 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__ 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/retry.py", line 56, in __call__ 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 77, in __call__ 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging self.on_deleted(obj) 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/policy.py", line 114, in on_deleted 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging self._drv_policy.release_network_policy(netpolicy_crd) 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 391, in release_network_policy 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging netpolicy_crd['metadata']['namespace']) 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 468, in _del_kuryrnetpolicy_crd 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging netpolicy_crd_name)) 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/k8s_client.py", line 132, in delete 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging raise exc.K8sClientException(response) 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging K8sClientException: <Response [404]> 2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging 2019-02-14 16:22:49.782 1 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/dispatch.py:62 2019-02-14 16:22:49.782 1 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Ignoring Kubernetes endpoints kube-controller-manager on_present /usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handler s/lbaas.py:251