Bug 1825169 - NP CRD patching broken due to CRD holding outdated data
Summary: NP CRD patching broken due to CRD holding outdated data
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.5
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.5.0
Assignee: Maysa Macedo
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-17 09:51 UTC by Maysa Macedo
Modified: 2020-07-13 17:28 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-13 17:28:32 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 211 0 None closed Bug 1825169: Ensure SG rule is deleted from CRD upon Namespace deletion 2020-06-19 11:02:42 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:28:54 UTC

Description Maysa Macedo 2020-04-17 09:51:58 UTC
Description of problem:

When a Namespace is updated the SG rule is first being deleted on Neutron, but not on the CRD, and when the the namespaces sg rules are created the CRD may be holding duplicated SG rules, but with different ids. Later if that Network policy is updated only the first repeated sg rule will get the ID and the second one will have no ID, causing the following error:

2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils [-] Error updating kuryrnetpolicy CRD np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector: kuryr_kubernetes.exceptions.K8sClientException: {"kind":"St
atus","apiVersion":"v1","metadata":{},"status":"Failure","message":"KuryrNetPolicy.openstack.org \"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector\" is invalid: spec.ingressSgRules.security_group_rule.id: Required value"
,"reason":"Invalid","details":{"name":"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector","group":"openstack.org","kind":"KuryrNetPolicy","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.in
gressSgRules.security_group_rule.id"}]},"code":422}
2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils Traceback (most recent call last):
2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/utils.py", line 236, in patch_kuryrnetworkpolicy_crd
2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils 'networkpolicy_spec': np_spec})
2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 139, in patch_crd
2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils self._raise_from_response(response)
2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 83, in _raise_from_response
2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils raise exc.K8sClientException(response.text)
2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils kuryr_kubernetes.exceptions.K8sClientException: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"KuryrNetPolicy.openstack.org \"np-all
ow-ns-b-via-namespace-selector-or-client-b-via-pod-selector\" is invalid: spec.ingressSgRules.security_group_rule.id: Required value","reason":"Invalid","details":{"name":"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector"
,"group":"openstack.org","kind":"KuryrNetPolicy","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.ingressSgRules.security_group_rule.id"}]},"code":422}
2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils
2020-04-13 10:54:17.860 1 ERROR kuryr_kubernetes.controller.drivers.utils
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry [-] Report handler unhealthy NetworkPolicyHandler: kuryr_kubernetes.exceptions.K8sClientException: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","messag
e":"KuryrNetPolicy.openstack.org \"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector\" is invalid: spec.ingressSgRules.security_group_rule.id: Required value","reason":"Invalid","details":{"name":"np-allow-ns-b-via-namespa
ce-selector-or-client-b-via-pod-selector","group":"openstack.org","kind":"KuryrNetPolicy","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.ingressSgRules.security_group_rule.id"}]},"code":422}
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry Traceback (most recent call last):
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry self._handler(event)
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 84, in __call__
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry self.on_present(obj)
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/policy.py", line 53, in on_present
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry project_id)
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 58, in ensure_network_policy
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry self.update_security_group_rules_from_network_policy(policy))
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 119, in update_security_group_rules_from_network_policy
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry np_spec=policy['spec'])
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/utils.py", line 236, in patch_kuryrnetworkpolicy_crd
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry 'networkpolicy_spec': np_spec})
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 139, in patch_crd
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry self._raise_from_response(response)
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 83, in _raise_from_response
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry raise exc.K8sClientException(response.text)
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry kuryr_kubernetes.exceptions.K8sClientException: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"KuryrNetPolicy.openstack.org \"np-allow-ns-b-vi
a-namespace-selector-or-client-b-via-pod-selector\" is invalid: spec.ingressSgRules.security_group_rule.id: Required value","reason":"Invalid","details":{"name":"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector","group":"
openstack.org","kind":"KuryrNetPolicy","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.ingressSgRules.security_group_rule.id"}]},"code":422}
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry
2020-04-13 10:54:17.862 1 ERROR kuryr_kubernetes.handlers.retry
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {'type': 'MODIFIED', 'object': {'kind': 'NetworkPolicy', 'apiVersion': 'networking.k8s.io/v1', 'metadata': {'name': 'allow-ns-b-via-namespace-sel
ector-or-client-b-via-pod-selector', 'namespace': 'network-policy-7344', 'selfLink': '/apis/networking.k8s.io/v1/namespaces/network-policy-7344/networkpolicies/allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector', 'uid': '50ca8
96d-ae95-45a5-8d2e-9534d050f5b4', 'resourceVersion': '2257159', 'generation': 1, 'creationTimestamp': '2020-04-13T10:54:07Z', 'annotations': {'kuryrnetpolicy_selfLink': '/apis/openstack.org/v1/namespaces/network-policy-7344/kuryrnetpolici
es/np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector'}, 'managedFields': [{'manager': 'e2e.test', 'operation': 'Update', 'apiVersion': 'networking.k8s.io/v1', 'time': '2020-04-13T10:54:07Z', 'fieldsType': 'FieldsV1', 'fiel
dsV1': {'f:spec': {'f:ingress': {}, 'f:podSelector': {'f:matchLabels': {'.': {}, 'f:pod-name': {}}}, 'f:policyTypes': {}}}}, {'manager': 'python-requests', 'operation': 'Update', 'apiVersion': 'networking.k8s.io/v1', 'time': '2020-04-13T1
0:54:13Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:kuryrnetpolicy_selfLink': {}}}}}]}, 'spec': {'podSelector': {'matchLabels': {'pod-name': 'server'}}, 'ingress': [{'from': [{'podSelector': {'ma
tchLabels': {'pod-name': 'client-b'}}}, {'namespaceSelector': {'matchLabels': {'ns-name': 'network-policy-b'}}}]}], 'policyTypes': ['Ingress']}}}: kuryr_kubernetes.exceptions.K8sClientException: {"kind":"Status","apiVersion":"v1","metadat
a":{},"status":"Failure","message":"KuryrNetPolicy.openstack.org \"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector\" is invalid: spec.ingressSgRules.security_group_rule.id: Required value","reason":"Invalid","details":{"
name":"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector","group":"openstack.org","kind":"KuryrNetPolicy","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.ingressSgRules.security_group_rule
.id"}]},"code":422}
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 84, in __call__
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging self.on_present(obj)
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/policy.py", line 53, in on_present
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging project_id)
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 58, in ensure_network_policy
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging self.update_security_group_rules_from_network_policy(policy))
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy.py", line 119, in update_security_group_rules_from_network_policy
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging np_spec=policy['spec'])
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/utils.py", line 236, in patch_kuryrnetworkpolicy_crd
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging 'networkpolicy_spec': np_spec})
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 139, in patch_crd
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging self._raise_from_response(response)
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 83, in _raise_from_response
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging raise exc.K8sClientException(response.text)
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging kuryr_kubernetes.exceptions.K8sClientException: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"KuryrNetPolicy.openstack.org \"np-allow-ns-b-
via-namespace-selector-or-client-b-via-pod-selector\" is invalid: spec.ingressSgRules.security_group_rule.id: Required value","reason":"Invalid","details":{"name":"np-allow-ns-b-via-namespace-selector-or-client-b-via-pod-selector","group"
:"openstack.org","kind":"KuryrNetPolicy","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.ingressSgRules.security_group_rule.id"}]},"code":422}
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging
2020-04-13 10:54:17.869 1 ERROR kuryr_kubernetes.handlers.logging

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Install OCP on OSP with Kuryr
2. Run Network Policy test number 6
3. Verify the Test fails and the Kuryr-controller restarts due to traceback shown above

Actual results:


Expected results:


Additional info:

Comment 3 Jon Uriarte 2020-04-30 14:45:00 UTC
Verified in 4.5.0-0.nightly-2020-04-29-144201 on top of RHOS_TRUNK-16.0-RHEL-8-20200427.n.0 OSP 16 compose.
OSP 16 deployment with OVS neutron backend.

OCP 4.5 installation with Kuryr completed successfully and NetworkPolicy-06 test case passed 3 times.
Kuryr-controller pod wasn't restarted during the execution of the test due to the traceback in the
description of this BZ.

Comment 4 errata-xmlrpc 2020-07-13 17:28:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.