Bug 1810517 - wrong attempt to retrieve pods list from kubernetes response
Summary: wrong attempt to retrieve pods list from kubernetes response
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.4
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: 4.5.0
Assignee: Maysa Macedo
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On: 1819129
Blocks: 1810521
TreeView+ depends on / blocked
 
Reported: 2020-03-05 12:28 UTC by Maysa Macedo
Modified: 2020-08-04 18:03 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1810521 (view as bug list)
Environment:
Last Closed: 2020-08-04 18:03:45 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 175 0 None closed Bug 1810517: Ensure list of pods items is retrieved 2020-06-23 09:02:12 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-08-04 18:03:49 UTC

Description Maysa Macedo 2020-03-05 12:28:20 UTC
Description of problem:


Right now we attempt to retrieve the list of pods without
fetching the items field returned on the Kubernetes response,
causing the wrong element to be fetched.

Traceback of the error:

2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/kuryrnet.py", line 87, in on_present
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry crd_selectors = self._drv_sg.update_namespace_sg_rules(namespace)
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 574, in update_namespace
_sg_rules
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry crd_selectors.extend(self.create_namespace_sg_rules(namespace))
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 557, in create_namespace
_sg_rules
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry 'ingress', crd, namespace=namespace)
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 369, in _parse_rules
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry rule_block, crd_rules, namespace, matched)
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 330, in _parse_selectors
_on_namespace
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry crd_rules, matched, crd))
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 159, in _create_sg_rule_
on_text_port
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry rule_selected_pod)
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 127, in _create_sg_rules
_with_container_ports
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry pod_ip = driver_utils.get_pod_ip(rule_selected_pod)
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "</usr/local/lib/python3.6/site-packages/decorator.py:decorator-gen-96>", line 2, in get_pod_ip
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/dogpile/cache/region.py", line 1360, in get_or_create_for_user_func
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry key, user_func, timeout, should_cache_fn, (arg, kw)
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/dogpile/cache/region.py", line 962, in get_or_create
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry async_creator,
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/dogpile/lock.py", line 187, in __enter__
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry return self._enter()
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/dogpile/lock.py", line 94, in _enter
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry generated = self._enter_create(value, createdtime)
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/dogpile/lock.py", line 180, in _enter_create
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry return self.creator()
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/dogpile/cache/region.py", line 916, in gen_value
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry *creator_args[0], **creator_args[1]
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/utils.py", line 292, in get_pod_ip
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry pod_metadata = pod['metadata']['annotations']
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry TypeError: string indices must be integers

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Run Network Policy test(number 13)
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 Jon Uriarte 2020-04-03 08:47:59 UTC
Verified in 4.5.0-0.nightly-2020-04-02-004321 on top of OSP 16 RHOS_TRUNK-16.0-RHEL-8-20200324.n.0 compose.

NetworkPolicy-13 test case passes, kuryr-controller pod is not restarted, and the traceback described in this BZ is not being thrown.

$ kubetest --provider=local  --check-version-skew=false --test --test_args="--ginkgo.focus=\[Feature:NetworkPolicy-13\] --host=https://api.ostest.shiftstack.com:6443"
....
{"msg":"PASSED [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should allow ingress access from namespace on one named port [Feature:NetworkPolicy-13]","total":1,"completed":1,"skipped":4785,"failed":0}
Apr  3 08:45:57.820: INFO: Running AfterSuite actions on all nodes
Apr  3 08:45:57.820: INFO: Running AfterSuite actions on node 1
{"msg":"Test Suite completed","total":1,"completed":1,"skipped":4842,"failed":0}

Ran 1 of 4843 Specs in 396.731 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 4842 Skipped

Comment 5 errata-xmlrpc 2020-08-04 18:03:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5 image release advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.