Bug 1792239 - NetworkPolicyHandler: KeyError when deleting a network policy
Summary: NetworkPolicyHandler: KeyError when deleting a network policy
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.4.0
Assignee: Luis Tomas Bolivar
QA Contact: GenadiC
URL:
Whiteboard:
Depends On:
Blocks: 1792240
TreeView+ depends on / blocked
 
Reported: 2020-01-17 11:06 UTC by Luis Tomas Bolivar
Modified: 2020-05-04 11:25 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1792240 (view as bug list)
Environment:
Last Closed: 2020-05-04 11:24:53 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 141 0 None closed Bug 1792239: Avoid KeyError when deleting NPs 2020-04-27 08:44:36 UTC
Launchpad 1860106 0 None None None 2020-01-17 11:06:43 UTC
OpenStack gerrit 703054 0 None MERGED Avoid KeyError when deleting NPs 2020-04-27 08:44:35 UTC
Red Hat Product Errata RHBA-2020:0581 0 None None None 2020-05-04 11:25:35 UTC

Description Luis Tomas Bolivar 2020-01-17 11:06:43 UTC
When having several network policies in parallel, and deleting one of them, the next KeyError may happen:
2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry [-] Report handler unhealthy NetworkPolicyHandler: KeyError: ('140f29c1-2980-405a-9572-6251477a658d', '47c1159c-5bba-44f5-b2a5-f6dae011f0db')
2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry Traceback (most recent call last):
2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__
2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry self._handler(event)
2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 77, in __call__
2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry self.on_deleted(obj)
2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/policy.py", line 130, in on_deleted
2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry self._drv_vif_pool.remove_sg_from_pools(crd_sg, net_id)
2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 1077, in remove_sg_from_pools
2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry vif_drv.remove_sg_from_pools(sg_id, net_id)
2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 313, in remove_sg_from_pools
2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry del self._available_ports_pools[pool_key][sg_key]
2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry KeyError: ('140f29c1-2980-405a-9572-6251477a658d', '47c1159c-5bba-44f5-b2a5-f6dae011f0db')
2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry
2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {'type': 'DELETED', 'object': {'kind': 'NetworkPolicy', 'apiVersion': 'networking.k8s.io/v1', 'metadata': {'name': 'all
ow-to-server-a-pod-selector', 'namespace': 'network-policy-6817', 'selfLink': '/apis/networking.k8s.io/v1/namespaces/network-policy-6817/networkpolicies/allow-to-server-a-pod-selector', 'uid': '7ab13f32-861e-45b9
-866d-9baa07b6ddd0', 'resourceVersion': '153310', 'generation': 1, 'creationTimestamp': '2020-01-16T17:03:25Z', 'annotations': {'kuryrnetpolicy_selfLink': '/apis/openstack.org/v1/namespaces/network-policy-6817/kuryrnetpolicies/np-allow-to-server-a-pod-selector'}}, 'spec': {'podSelector': {'matchLabels': {'pod-name': 'client-a'}}, 'egress': [{'ports': [{'protocol': 'UDP', 'port': 53}]}, {'to': [{'podSelector': {'matchLabe
ls': {'pod-name': 'server'}}}]}], 'policyTypes': ['Egress']}}}: KeyError: ('140f29c1-2980-405a-9572-6251477a658d', '47c1159c-5bba-44f5-b2a5-f6dae011f0db')
2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__
2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__
2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 77, in __call__
2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging self.on_deleted(obj)
2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/policy.py", line 130, in on_deleted
2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging self._drv_vif_pool.remove_sg_from_pools(crd_sg, net_id)
2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 1077, in remove_sg_from_pools
2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging vif_drv.remove_sg_from_pools(sg_id, net_id)
2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 313, in remove_sg_from_pools
2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging del self._available_ports_pools[pool_key][sg_key]
2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging KeyError: ('140f29c1-2980-405a-9572-6251477a658d', '47c1159c-5bba-44f5-b2a5-f6dae011f0db')

Comment 2 Jon Uriarte 2020-01-28 11:29:18 UTC
Verified in 4.4.0-0.nightly-2020-01-24-141203 build on top of OSP 16 RHOS_TRUNK-16.0-RHEL-8-20191224.n.0 puddle.

The OCP installer finishes successfully:

 $ oc get clusterversion
 NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
 version   4.4.0-0.nightly-2020-01-24-141203   True        False         8m21s   Cluster version is 4.4.0-0.nightly-2020-01-24-141203

After running several K8s Network Policy tests in parallel (creating/deleting policies) the KeyError message was not shown in Kuryr
controller pod.

Comment 4 errata-xmlrpc 2020-05-04 11:24:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.