Description of problem: When a Namespace is updated the SG rule is first being deleted on Neutron, but not on the CRD, and when the the namespaces sg rules are created the CRD may be holding duplicated SG rules, but with different ids. Later if that Network policy is updated only the first repeated sg rule will get the ID and the second one will have no ID, causing the following error: 2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils [-] Error updating kuryrnetpolicy CRD np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector: kuryr_kubernetes.exceptions.K8sClientException: {"kind":"St atus","apiVersion":"v1","metadata":{},"status":"Failure","message":"KuryrNetPolicy.openstack.org \"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector\" is invalid: spec.ingressSgRules.security_group_rule.id: Required value" ,"reason":"Invalid","details":{"name":"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector","group":"openstack.org","kind":"KuryrNetPolicy","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.in gressSgRules.security_group_rule.id"}]},"code":422} 2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils Traceback (most recent call last): 2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/utils.py", line 236, in patch_kuryrnetworkpolicy_crd 2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils 'networkpolicy_spec': np_spec}) 2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 139, in patch_crd 2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils self._raise_from_response(response) 2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 83, in _raise_from_response 2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils raise exc.K8sClientException(response.text) 2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils kuryr_kubernetes.exceptions.K8sClientException: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"KuryrNetPolicy.openstack.org \"np-all ow-ns-b-via-namespace-selector-or-client-b-via-pod-selector\" is invalid: spec.ingressSgRules.security_group_rule.id: Required value","reason":"Invalid","details":{"name":"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector" ,"group":"openstack.org","kind":"KuryrNetPolicy","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.ingressSgRules.security_group_rule.id"}]},"code":422} 2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils 2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry [-] Report handler unhealthy NetworkPolicyHandler: kuryr_kubernetes.exceptions.K8sClientException: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","messag e":"KuryrNetPolicy.openstack.org \"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector\" is invalid: spec.ingressSgRules.security_group_rule.id: Required value","reason":"Invalid","details":{"name":"np-allow-ns-b-via-namespa ce-selector-or-client-b-via-pod-selector","group":"openstack.org","kind":"KuryrNetPolicy","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.ingressSgRules.security_group_rule.id"}]},"code":422} 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry Traceback (most recent call last): 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__ 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry self._handler(event) 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 84, in __call__ 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry self.on_present(obj) 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/policy.py", line 53, in on_present 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry project_id) 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 58, in ensure_network_policy 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry self.update_security_group_rules_from_network_policy(policy)) 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 119, in update_security_group_rules_from_network_policy 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry np_spec=policy['spec']) 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/utils.py", line 236, in patch_kuryrnetworkpolicy_crd 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry 'networkpolicy_spec': np_spec}) 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 139, in patch_crd 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry self._raise_from_response(response) 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 83, in _raise_from_response 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry raise exc.K8sClientException(response.text) 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry kuryr_kubernetes.exceptions.K8sClientException: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"KuryrNetPolicy.openstack.org \"np-allow-ns-b-vi a-namespace-selector-or-client-b-via-pod-selector\" is invalid: spec.ingressSgRules.security_group_rule.id: Required value","reason":"Invalid","details":{"name":"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector","group":" openstack.org","kind":"KuryrNetPolicy","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.ingressSgRules.security_group_rule.id"}]},"code":422} 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry 2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {'type': 'MODIFIED', 'object': {'kind': 'NetworkPolicy', 'apiVersion': 'networking.k8s.io/v1', 'metadata': {'name': 'allow-ns-b-via-namespace-sel ector-or-client-b-via-pod-selector', 'namespace': 'network-policy-7344', 'selfLink': '/apis/networking.k8s.io/v1/namespaces/network-policy-7344/networkpolicies/allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector', 'uid': '50ca8 96d-ae95-45a5-8d2e-9534d050f5b4', 'resourceVersion': '2257159', 'generation': 1, 'creationTimestamp': '2020-04-13T10:54:07Z', 'annotations': {'kuryrnetpolicy_selfLink': '/apis/openstack.org/v1/namespaces/network-policy-7344/kuryrnetpolici es/np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector'}, 'managedFields': [{'manager': 'e2e.test', 'operation': 'Update', 'apiVersion': 'networking.k8s.io/v1', 'time': '2020-04-13T10:54:07Z', 'fieldsType': 'FieldsV1', 'fiel dsV1': {'f:spec': {'f:ingress': {}, 'f:podSelector': {'f:matchLabels': {'.': {}, 'f:pod-name': {}}}, 'f:policyTypes': {}}}}, {'manager': 'python-requests', 'operation': 'Update', 'apiVersion': 'networking.k8s.io/v1', 'time': '2020-04-13T1 0:54:13Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:kuryrnetpolicy_selfLink': {}}}}}]}, 'spec': {'podSelector': {'matchLabels': {'pod-name': 'server'}}, 'ingress': [{'from': [{'podSelector': {'ma tchLabels': {'pod-name': 'client-b'}}}, {'namespaceSelector': {'matchLabels': {'ns-name': 'network-policy-b'}}}]}], 'policyTypes': ['Ingress']}}}: kuryr_kubernetes.exceptions.K8sClientException: {"kind":"Status","apiVersion":"v1","metadat a":{},"status":"Failure","message":"KuryrNetPolicy.openstack.org \"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector\" is invalid: spec.ingressSgRules.security_group_rule.id: Required value","reason":"Invalid","details":{" name":"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector","group":"openstack.org","kind":"KuryrNetPolicy","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.ingressSgRules.security_group_rule .id"}]},"code":422} 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last): 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__ 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__ 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 84, in __call__ 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging self.on_present(obj) 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/policy.py", line 53, in on_present 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging project_id) 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 58, in ensure_network_policy 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging self.update_security_group_rules_from_network_policy(policy)) 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 119, in update_security_group_rules_from_network_policy 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging np_spec=policy['spec']) 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/utils.py", line 236, in patch_kuryrnetworkpolicy_crd 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging 'networkpolicy_spec': np_spec}) 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 139, in patch_crd 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging self._raise_from_response(response) 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 83, in _raise_from_response 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging raise exc.K8sClientException(response.text) 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging kuryr_kubernetes.exceptions.K8sClientException: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"KuryrNetPolicy.openstack.org \"np-allow-ns-b- via-namespace-selector-or-client-b-via-pod-selector\" is invalid: spec.ingressSgRules.security_group_rule.id: Required value","reason":"Invalid","details":{"name":"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector","group" :"openstack.org","kind":"KuryrNetPolicy","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.ingressSgRules.security_group_rule.id"}]},"code":422} 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging 2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Install OCP on OSP with Kuryr 2. Run Network Policy test number 6 3. Verify the Test fails and the Kuryr-controller restarts due to traceback shown above Actual results: Expected results: Additional info:
Verified in 4.5.0-0.nightly-2020-04-29-144201 on top of RHOS_TRUNK-16.0-RHEL-8-20200427.n.0 OSP 16 compose. OSP 16 deployment with OVS neutron backend. OCP 4.5 installation with Kuryr completed successfully and NetworkPolicy-06 test case passed 3 times. Kuryr-controller pod wasn't restarted during the execution of the test due to the traceback in the description of this BZ.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409