Bug 1875751 - "Resource not found" if NP deleted during processing
Summary: "Resource not found" if NP deleted during processing
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.6
Hardware: All
OS: All
unspecified
medium
Target Milestone: ---
: 4.6.0
Assignee: Michał Dulko
QA Contact: GenadiC
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-04 09:14 UTC by Michał Dulko
Modified: 2020-09-10 11:51 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)
NP test results (189.50 KB, application/gzip)
2020-09-10 11:51 UTC, rlobillo
no flags Details


Links
System ID Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 340 None closed Bug 1875751: NP: Protect from disappearing resources 2020-09-09 07:32:11 UTC

Description Michał Dulko 2020-09-04 09:14:08 UTC
Description of problem:
  We have various issues in NP code when resources are listed and while
  processed one gets removed. E.g.:

  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry [-] Report handler unhealthy KuryrPortHandler: kuryr_kubernetes.exceptions.K8sResourceNotFound: Resource not found: '{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"networkpolicies.networking.k8s.io \\"allow-from-client-a-pod-selector\\" not found","reason":"NotFound","details":{"name":"allow-from-client-a-pod-selector","group":"networking.k8s.io","kind":"networkpolicies"},"code":404}\n'
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry Traceback (most recent call last):
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry     self._handler(event, *args, **kwargs)
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 84, in __call__
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry     self.on_present(obj)
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/kuryrport.py", line 126, in on_present
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry     crd_pod_selectors = self._drv_sg.create_sg_rules(pod)
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 324, in create_sg_rules
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry     _bump_networkpolicy(crd)
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 53, in _bump_networkpolicy
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry     {constants.K8S_ANNOTATION_POLICY: str(uuid.uuid4())})
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 368, in annotate
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry     self._raise_from_response(response)
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 83, in _raise_from_response
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry     raise exc.K8sResourceNotFound(response.text)
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry kuryr_kubernetes.exceptions.K8sResourceNotFound: Resource not found: '{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"networkpolicies.networking.k8s.io \\"allow-from-client-a-pod-selector\\" not found","reason":"NotFound","details":{"name":"allow-from-client-a-pod-selector","group":"networking.k8s.io","kind":"networkpolicies"},"code":404}\n'

Version-Release number of selected component (if applicable):


How reproducible:
Quite a race condition.

Steps to Reproduce:
1. This happened when running network policy e2e tests, so that's the best we have.

Actual results:
Error causes kuryr-controller restart.

Expected results:
Errors ignored, no trace of them in the logs, kuryr-controller not restarted.

Additional info:

Comment 3 rlobillo 2020-09-10 11:39:19 UTC
Verified on OCP4.6.0-0.nightly-2020-09-10-011413 over OSP16.1 (RHOS-16.1-RHEL-8-20200831.n.1) with OVN-Octavia.

All NP tests PASSED. No kuryr-controller restarts.

Logs attached.

Comment 4 rlobillo 2020-09-10 11:51:17 UTC
Created attachment 1714420 [details]
NP test results


Note You need to log in before you can comment on or make changes to this bug.