Description of problem:
  We have various issues in NP code when resources are listed and while
  processed one gets removed. E.g.:

  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry [-] Report handler unhealthy KuryrPortHandler: kuryr_kubernetes.exceptions.K8sResourceNotFound: Resource not found: '{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"networkpolicies.networking.k8s.io \\"allow-from-client-a-pod-selector\\" not found","reason":"NotFound","details":{"name":"allow-from-client-a-pod-selector","group":"networking.k8s.io","kind":"networkpolicies"},"code":404}\n'
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry Traceback (most recent call last):
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry     self._handler(event, *args, **kwargs)
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 84, in __call__
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry     self.on_present(obj)
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/kuryrport.py", line 126, in on_present
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry     crd_pod_selectors = self._drv_sg.create_sg_rules(pod)
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 324, in create_sg_rules
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry     _bump_networkpolicy(crd)
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 53, in _bump_networkpolicy
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry     {constants.K8S_ANNOTATION_POLICY: str(uuid.uuid4())})
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 368, in annotate
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry     self._raise_from_response(response)
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 83, in _raise_from_response
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry     raise exc.K8sResourceNotFound(response.text)
  2020-09-03 11:36:01.294 1 ERROR kuryr_kubernetes.handlers.retry kuryr_kubernetes.exceptions.K8sResourceNotFound: Resource not found: '{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"networkpolicies.networking.k8s.io \\"allow-from-client-a-pod-selector\\" not found","reason":"NotFound","details":{"name":"allow-from-client-a-pod-selector","group":"networking.k8s.io","kind":"networkpolicies"},"code":404}\n'

Version-Release number of selected component (if applicable):


How reproducible:
Quite a race condition.

Steps to Reproduce:
1. This happened when running network policy e2e tests, so that's the best we have.

Actual results:
Error causes kuryr-controller restart.

Expected results:
Errors ignored, no trace of them in the logs, kuryr-controller not restarted.

Additional info: