Bug 1792239
| Summary: | NetworkPolicyHandler: KeyError when deleting a network policy | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Luis Tomas Bolivar <ltomasbo> | |
| Component: | Networking | Assignee: | Luis Tomas Bolivar <ltomasbo> | |
| Networking sub component: | kuryr | QA Contact: | GenadiC <gcheresh> | |
| Status: | CLOSED ERRATA | Docs Contact: | ||
| Severity: | medium | |||
| Priority: | medium | CC: | juriarte | |
| Version: | 4.4 | |||
| Target Milestone: | --- | |||
| Target Release: | 4.4.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | No Doc Update | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1792240 (view as bug list) | Environment: | ||
| Last Closed: | 2020-05-04 11:24:53 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1792240 | |||
Verified in 4.4.0-0.nightly-2020-01-24-141203 build on top of OSP 16 RHOS_TRUNK-16.0-RHEL-8-20191224.n.0 puddle. The OCP installer finishes successfully: $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.0-0.nightly-2020-01-24-141203 True False 8m21s Cluster version is 4.4.0-0.nightly-2020-01-24-141203 After running several K8s Network Policy tests in parallel (creating/deleting policies) the KeyError message was not shown in Kuryr controller pod. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581 |
When having several network policies in parallel, and deleting one of them, the next KeyError may happen: 2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry [-] Report handler unhealthy NetworkPolicyHandler: KeyError: ('140f29c1-2980-405a-9572-6251477a658d', '47c1159c-5bba-44f5-b2a5-f6dae011f0db') 2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry Traceback (most recent call last): 2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__ 2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry self._handler(event) 2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 77, in __call__ 2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry self.on_deleted(obj) 2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/policy.py", line 130, in on_deleted 2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry self._drv_vif_pool.remove_sg_from_pools(crd_sg, net_id) 2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 1077, in remove_sg_from_pools 2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry vif_drv.remove_sg_from_pools(sg_id, net_id) 2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 313, in remove_sg_from_pools 2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry del self._available_ports_pools[pool_key][sg_key] 2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry KeyError: ('140f29c1-2980-405a-9572-6251477a658d', '47c1159c-5bba-44f5-b2a5-f6dae011f0db') 2020-01-16 17:08:48.629 1 ERROR kuryr_kubernetes.handlers.retry 2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {'type': 'DELETED', 'object': {'kind': 'NetworkPolicy', 'apiVersion': 'networking.k8s.io/v1', 'metadata': {'name': 'all ow-to-server-a-pod-selector', 'namespace': 'network-policy-6817', 'selfLink': '/apis/networking.k8s.io/v1/namespaces/network-policy-6817/networkpolicies/allow-to-server-a-pod-selector', 'uid': '7ab13f32-861e-45b9 -866d-9baa07b6ddd0', 'resourceVersion': '153310', 'generation': 1, 'creationTimestamp': '2020-01-16T17:03:25Z', 'annotations': {'kuryrnetpolicy_selfLink': '/apis/openstack.org/v1/namespaces/network-policy-6817/kuryrnetpolicies/np-allow-to-server-a-pod-selector'}}, 'spec': {'podSelector': {'matchLabels': {'pod-name': 'client-a'}}, 'egress': [{'ports': [{'protocol': 'UDP', 'port': 53}]}, {'to': [{'podSelector': {'matchLabe ls': {'pod-name': 'server'}}}]}], 'policyTypes': ['Egress']}}}: KeyError: ('140f29c1-2980-405a-9572-6251477a658d', '47c1159c-5bba-44f5-b2a5-f6dae011f0db') 2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last): 2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__ 2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) 2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__ 2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) 2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 77, in __call__ 2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging self.on_deleted(obj) 2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/policy.py", line 130, in on_deleted 2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging self._drv_vif_pool.remove_sg_from_pools(crd_sg, net_id) 2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 1077, in remove_sg_from_pools 2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging vif_drv.remove_sg_from_pools(sg_id, net_id) 2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 313, in remove_sg_from_pools 2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging del self._available_ports_pools[pool_key][sg_key] 2020-01-16 17:08:48.637 1 ERROR kuryr_kubernetes.handlers.logging KeyError: ('140f29c1-2980-405a-9572-6251477a658d', '47c1159c-5bba-44f5-b2a5-f6dae011f0db')