Bug 1810517

Summary:	wrong attempt to retrieve pods list from kubernetes response
Product:	OpenShift Container Platform	Reporter:	Maysa Macedo <mdemaced>
Component:	Networking	Assignee:	Maysa Macedo <mdemaced>
Networking sub component:	kuryr	QA Contact:	Jon Uriarte <juriarte>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	low
Priority:	medium	CC:	juriarte, ltomasbo
Version:	4.4
Target Milestone:	---
Target Release:	4.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1810521 (view as bug list)		Environment:
Last Closed:	2020-08-04 18:03:45 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1819129
Bug Blocks:	1810521

Description Maysa Macedo 2020-03-05 12:28:20 UTC

Description of problem:


Right now we attempt to retrieve the list of pods without
fetching the items field returned on the Kubernetes response,
causing the wrong element to be fetched.

Traceback of the error:

2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/kuryrnet.py", line 87, in on_present
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry crd_selectors = self._drv_sg.update_namespace_sg_rules(namespace)
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 574, in update_namespace
_sg_rules
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry crd_selectors.extend(self.create_namespace_sg_rules(namespace))
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 557, in create_namespace
_sg_rules
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry 'ingress', crd, namespace=namespace)
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 369, in _parse_rules
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry rule_block, crd_rules, namespace, matched)
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 330, in _parse_selectors
_on_namespace
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry crd_rules, matched, crd))
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 159, in _create_sg_rule_
on_text_port
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry rule_selected_pod)
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/network_policy_security_groups.py", line 127, in _create_sg_rules
_with_container_ports
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry pod_ip = driver_utils.get_pod_ip(rule_selected_pod)
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "</usr/local/lib/python3.6/site-packages/decorator.py:decorator-gen-96>", line 2, in get_pod_ip
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/dogpile/cache/region.py", line 1360, in get_or_create_for_user_func
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry key, user_func, timeout, should_cache_fn, (arg, kw)
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/dogpile/cache/region.py", line 962, in get_or_create
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry async_creator,
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/dogpile/lock.py", line 187, in __enter__
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry return self._enter()
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/dogpile/lock.py", line 94, in _enter
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry generated = self._enter_create(value, createdtime)
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/dogpile/lock.py", line 180, in _enter_create
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry return self.creator()
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/dogpile/cache/region.py", line 916, in gen_value
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry *creator_args[0], **creator_args[1]
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/utils.py", line 292, in get_pod_ip
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry pod_metadata = pod['metadata']['annotations']
2020-02-24 07:43:18.163 1 ERROR kuryr_kubernetes.handlers.retry TypeError: string indices must be integers

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Run Network Policy test(number 13)
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 Jon Uriarte 2020-04-03 08:47:59 UTC

Verified in 4.5.0-0.nightly-2020-04-02-004321 on top of OSP 16 RHOS_TRUNK-16.0-RHEL-8-20200324.n.0 compose.

NetworkPolicy-13 test case passes, kuryr-controller pod is not restarted, and the traceback described in this BZ is not being thrown.

$ kubetest --provider=local  --check-version-skew=false --test --test_args="--ginkgo.focus=\[Feature:NetworkPolicy-13\] --host=https://api.ostest.shiftstack.com:6443"
....
{"msg":"PASSED [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should allow ingress access from namespace on one named port [Feature:NetworkPolicy-13]","total":1,"completed":1,"skipped":4785,"failed":0}
Apr  3 08:45:57.820: INFO: Running AfterSuite actions on all nodes
Apr  3 08:45:57.820: INFO: Running AfterSuite actions on node 1
{"msg":"Test Suite completed","total":1,"completed":1,"skipped":4842,"failed":0}

Ran 1 of 4843 Specs in 396.731 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 4842 Skipped

Comment 5 errata-xmlrpc 2020-08-04 18:03:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5 image release advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409