Bug 1677562 - Race when deleting namespaces with network policies
Summary: Race when deleting namespaces with network policies
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-kuryr-kubernetes
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: Upstream M3
: 15.0 (Stein)
Assignee: Luis Tomas Bolivar
QA Contact: GenadiC
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-15 09:07 UTC by Luis Tomas Bolivar
Modified: 2019-09-26 10:48 UTC (History)
1 user (show)

Fixed In Version: openstack-kuryr-kubernetes-0.6.2-0.20190305141049.a019712.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 11:20:21 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1816020 0 None None None 2019-02-15 09:07:37 UTC
OpenStack gerrit 637153 0 None None None 2019-02-19 14:31:21 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:20:38 UTC

Description Luis Tomas Bolivar 2019-02-15 09:07:37 UTC 
When using network policies, there seems to be a race when deleting a namespace that contains pods, services and network policies.

When that happens, the next error appears on the kuryr-controller, making it failing the healtchecks and therefore forcing its restart:

2019-02-14 16:22:48.880 1 DEBUG kuryr_kubernetes.k8s_client [-] Delete /apis/openstack.org/v1/namespaces/test/kuryrnetpolicies/np-np delete /usr/lib/python2.7/site-packages/kuryr_kubernetes/k8s_client.py:122
2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy [-] Kubernetes Client Exception deleting kuryrnetpolicy CRD.: K8sClientException: <Response [404]>
2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy Traceback (most recent call last):
2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 468, in _del_kuryrnetpolicy
_crd
2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy netpolicy_crd_name))
2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/k8s_client.py", line 132, in delete
2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy raise exc.K8sClientException(response)
2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy K8sClientException: <Response [404]>
2019-02-14 16:22:48.906 1 ERROR kuryr_kubernetes.controller.drivers.network_policy
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry [-] Report handler unhealthy NetworkPolicyHandler: K8sClientException: <Response [404]>
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry Traceback (most recent call last):
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/retry.py", line 56, in __call__
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry self._handler(event)
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 77, in __call__
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry self.on_deleted(obj)
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/policy.py", line 114, in on_deleted
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry self._drv_policy.release_network_policy(netpolicy_crd)
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 391, in release_network_policy
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry netpolicy_crd['metadata']['namespace'])
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 468, in _del_kuryrnetpolicy_crd
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry netpolicy_crd_name))
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/k8s_client.py", line 132, in delete
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry raise exc.K8sClientException(response)
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry K8sClientException: <Response [404]>
2019-02-14 16:22:48.908 1 ERROR kuryr_kubernetes.handlers.retry
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {u'object': {u'kind': u'NetworkPolicy', u'spec': {u'policyTypes': [u'Ingress'], u'ingress': [{u'from': [{u'podSelector'
: {u'matchLabels': {u'run': u'no-demo'}}}], u'ports': [{u'protocol': u'TCP', u'port': 8080}]}], u'podSelector': {u'matchLabels': {u'run': u'demo'}}}, u'apiVersion': u'networking.k8s.io/v1', u'metadata': {u'name':
 u'np', u'generation': 1, u'namespace': u'test', u'resourceVersion': u'12504', u'creationTimestamp': u'2019-02-14T15:43:38Z', u'annotations': {u'kuryrnetpolicy_selfLink': u'/apis/openstack.org/v1/namespaces/test/
kuryrnetpolicies/np-np', u'kubectl.kubernetes.io/last-applied-configuration': u'{"apiVersion":"networking.k8s.io/v1","kind":"NetworkPolicy","metadata":{"annotations":{},"name":"np","namespace":"test"},"spec":{"in
gress":[{"from":[{"podSelector":{"matchLabels":{"run":"no-demo"}}}],"ports":[{"port":8080,"protocol":"TCP"}]}],"podSelector":{"matchLabels":{"run":"demo"}},"policyTypes":["Ingress"]}}\n'}, u'selfLink': u'/apis/ne
tworking.k8s.io/v1/namespaces/test/networkpolicies/np', u'uid': u'4c2447ec-306f-11e9-b3e0-842b2b2d5107'}}, u'type': u'DELETED'}: K8sClientException: <Response [404]>
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/retry.py", line 56, in __call__
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 77, in __call__
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging self.on_deleted(obj)
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/policy.py", line 114, in on_deleted
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging self._drv_policy.release_network_policy(netpolicy_crd)
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 391, in release_network_policy
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging netpolicy_crd['metadata']['namespace'])
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 468, in _del_kuryrnetpolicy_crd
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging netpolicy_crd_name))
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/k8s_client.py", line 132, in delete
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging raise exc.K8sClientException(response)
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging K8sClientException: <Response [404]>
2019-02-14 16:22:48.909 1 ERROR kuryr_kubernetes.handlers.logging
2019-02-14 16:22:49.782 1 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/dispatch.py:62
2019-02-14 16:22:49.782 1 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Ignoring Kubernetes endpoints kube-controller-manager on_present /usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handler
s/lbaas.py:251
Comment 5 errata-xmlrpc 2019-09-21 11:20:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811
Note You need to log in before you can comment on or make changes to this bug.