1860311 – kuryr-controller logs are awfully unreadable

Bug 1860311 - kuryr-controller logs are awfully unreadable

Summary: kuryr-controller logs are awfully unreadable

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Networking
Sub Component:
Version:	4.6
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	4.5.z
Assignee:	Michał Dulko
QA Contact:	GenadiC
Docs Contact:
URL:
Whiteboard:
Depends On:	1860030
Blocks:
TreeView+	depends on / blocked

Reported:	2020-07-24 09:51 UTC by Luis Tomas Bolivar
Modified:	2020-10-19 14:54 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	1860030
Environment:
Last Closed:	2020-10-19 14:54:24 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	openshift kuryr-kubernetes pull 312	None	closed	[release-4.5] Bug 1860311: Civilize logging	2020-12-21 03:42:51 UTC
Github	openshift kuryr-kubernetes pull 351	None	closed	Bug 1860311: Civilize logging vol 2	2020-12-21 03:42:53 UTC
Red Hat Product Errata	RHBA-2020:4228	None	None	None	2020-10-19 14:54:40 UTC

Comment 3 rlobillo 2020-08-21 12:24:35 UTC

There are still some tracebacks that are spamming the logs as indicated on Bug #1860030. 

2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     chunked=chunked)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._validate_conn(conn)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     conn.connect()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 344, in connect
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     ssl_context=context)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 354, in ssl_wrap_socket
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return context.wrap_socket(sock, server_hostname=server_hostname)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return GreenSSLSocket(sock, *a, _context=self, **kw)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     super(GreenSSLSocket, self).do_handshake)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return func(*a, **kw)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._sslobj.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._sslobj.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging OSError: [Errno 0] Error
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging 
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging During handling of the above exception, another exception occurred:
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging 
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     timeout=timeout
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     _stacktrace=sys.exc_info()[2])
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 368, in increment
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     raise six.reraise(type(error), error, _stacktrace)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 692, in reraise
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     raise value.with_traceback(tb)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     chunked=chunked)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._validate_conn(conn)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     conn.connect()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 344, in connect
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     ssl_context=context)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 354, in ssl_wrap_socket
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return context.wrap_socket(sock, server_hostname=server_hostname)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return GreenSSLSocket(sock, *a, _context=self, **kw)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     super(GreenSSLSocket, self).do_handshake)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return func(*a, **kw)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._sslobj.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._sslobj.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging urllib3.exceptions.ProtocolError: ('Connection aborted.', OSError(0, 'Error'))
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging 
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging During handling of the above exception, another exception occurred:
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging 
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._handler(event)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 67, in __call__
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._k8s.get(obj_link)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 81, in get
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     headers=header)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/api.py", line 75, in get
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return request('get', url, params=params, **kwargs)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/api.py", line 60, in request
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return session.request(method=method, url=url, **kwargs)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     resp = self.send(prep, **send_kwargs)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     r = adapter.send(request, **kwargs)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     raise ConnectionError(err, request=request)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging requests.exceptions.ConnectionError: ('Connection aborted.', OSError(0, 'Error'))
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging 
2020-08-17 10:40:21.253 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {'type': 'ADDED', 'object': {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'openshift-cluster-node-tuning-operator', 'selfLink': '/api/v1/namespaces/openshift-cluster-node-tuning-operator', 'uid': 'df907325-b35a-45be-8062-c39e86fbc05e', 'resourceVersion': '10837', 'creationTimestamp': '2020-08-17T10:13:22Z', 'annotations': {'openshift.io/node-selector': '', 'openshift.io/sa.scc.mcs': 's0:c21,c15', 'openshift.io/sa.scc.supplemental-groups': '1000450000/10000', 'openshift.io/sa.scc.uid-range': '1000450000/10000', 'openstack.org/kuryr-net-crd': 'ns-openshift-cluster-node-tuning-operator'}}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}}}: requests.exceptions.ConnectionError: ('Connection aborted.', OSError(0, 'Error'))

Comment 6 Itzik Brown 2020-10-10 23:58:46 UTC

Verified with 4.5.0-0.nightly-2020-10-09-125103

Comment 9 errata-xmlrpc 2020-10-19 14:54:24 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5.15 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4228

Note You need to log in before you can comment on or make changes to this bug.