Bug 1787341
| Summary: | Deletion of Network Policies enforced on same pod cause controller restart | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Maysa Macedo <mdemaced> | |
| Component: | Networking | Assignee: | Maysa Macedo <mdemaced> | |
| Networking sub component: | kuryr | QA Contact: | Jon Uriarte <juriarte> | |
| Status: | CLOSED ERRATA | Docs Contact: | ||
| Severity: | unspecified | |||
| Priority: | unspecified | CC: | juriarte, ltomasbo | |
| Version: | 4.4 | |||
| Target Milestone: | --- | |||
| Target Release: | 4.4.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1787343 (view as bug list) | Environment: | ||
| Last Closed: | 2020-05-04 11:22:00 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1787343 | |||
Verified in 4.4.0-0.nightly-2020-01-24-045907 build on top of OSP 13 2020-01-15.3 puddle.
The OCP installer finishes successfully:
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.4.0-0.nightly-2020-01-24-045907 True False 137m Cluster version is 4.4.0-0.nightly-2020-01-24-045907
The K8s Network Policy test "should enforce policies to check ingress and egress policies can be controlled independently based on PodSelector [Feature:NetworkPolicy-23]" has passed
two times and Kuryr controller has not been restarted.
• [SLOW TEST:511.191 seconds]
[sig-network] NetworkPolicy [LinuxOnly]
/home/stack/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/framework.go:23
NetworkPolicy between server and client
/home/stack/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/network_policy.go:56
should enforce policies to check ingress and egress policies can be controlled independently based on PodSelector [Feature:NetworkPolicy-23]
/home/stack/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/network_policy.go:1282
------------------------------
{"msg":"PASSED [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce policies to check ingress and egress policies can be controlled independently based on PodSelector [Feature:NetworkPolicy-23]","total":1,"completed":1,"skipped":1104,"failed":0}
Ran 1 of 4846 Specs in 511.324 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 4845 Skipped
PASS
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581 |
Description of problem: When multiple Network policies deletion affecting the same pod happens at the same time, a Not Found exception can be raised causing a Kuryr Controller restart. 2019-12-17 01:47:47.588 1 DEBUG kuryr_kubernetes.handlers.asynchronous [-] Asynchronous handler stopped processing group 05e19cdf-206e-11ea-9993-fa163e044615 _done /usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/asynchron ous.py:102 2019-12-17 01:47:47.749 1 DEBUG kuryr_kubernetes.controller.drivers.network_policy [-] Deleting KuryrNetPolicy CRD np-allow-pod-a-to-pod-b-using-pod-selector _del_kuryrnetpolicy_crd /usr/local/lib/python3.6/site-packages/kuryr_kubernetes/ controller/drivers/network_policy.py:740 2019-12-17 01:47:47.749 1 DEBUG kuryr_kubernetes.k8s_client [-] Delete /apis/openstack.org/v1/namespaces/network-policy-9919/kuryrnetpolicies/np-allow-pod-a-to-pod-b-using-pod-selector delete /usr/local/lib/python3.6/site-packages/kuryr_k ubernetes/k8s_client.py:185 2019-12-17 01:47:47.878 1 DEBUG kuryr_kubernetes.handlers.asynchronous [-] Asynchronous handler started processing f5f9040f-206e-11ea-9993-fa163e044615 _run /usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/asynchronous.py: 64 2019-12-17 01:47:47.906 1 DEBUG neutronclient.v2_0.client [-] Error message: {"NeutronError": {"message": "Security group 669aff8c-852b-4260-b04c-95ccbfb1a998 does not exist", "type": "SecurityGroupNotFound", "detail": ""}} _handle_fault_ response /usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py:259 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry [-] Report handler unhealthy NetworkPolicyHandler: neutronclient.common.exceptions.NotFound: Security group 669aff8c-852b-4260-b04c-95ccbfb1a998 does not exist Neutron server returns request_ids: ['req-b90ea3f9-3ee3-46a8-b526-3fccce6494f8'] 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry Traceback (most recent call last): 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__ 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry self._handler(event) 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 77, in __call__ 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry self.on_deleted(obj) 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/policy.py", line 119, in on_deleted 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry self._drv_vif_pool.update_vif_sgs(pod, pod_sgs) 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 1131, in update_vif_sgs 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry self._vif_drvs[pod_vif_type].update_vif_sgs(pod, sgs) 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 173, in update_vif_sgs 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry self._drv_vif.update_vif_sgs(pod, sgs) 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/neutron_vif.py", line 112, in update_vif_sgs 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry security_groups) 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 808, in update_port 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry revision_number=revision_number) 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 2399, in _update_resource 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry return self.put(path, **kwargs) 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 363, in put 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry headers=headers, params=params) 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry headers=headers, params=params) 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 294, in do_request 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry self._handle_fault_response(status_code, replybody, resp) 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry exception_handler_v20(status_code, error_body) 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry request_ids=request_ids) 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry neutronclient.common.exceptions.NotFound: Security group 669aff8c-852b-4260-b04c-95ccbfb1a998 does not exist 2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry Neutron server returns request_ids: ['req-b90ea3f9-3ee3-46a8-b526-3fccce6494f8'] Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Run the following Kubernetes Network Policy test: "should enforce policies to check ingress and egress policies can be controlled independently based on PodSelector" 2. 3. Actual results: Kuryr Controller restarted Expected results: Both Network Policies and respective security groups are deleted with no controller restart. Additional info: