Bug 1860311 - kuryr-controller logs are awfully unreadable
Summary: kuryr-controller logs are awfully unreadable
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.5.z
Assignee: Michał Dulko
QA Contact: GenadiC
URL:
Whiteboard:
Depends On: 1860030
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-24 09:51 UTC by Luis Tomas Bolivar
Modified: 2020-10-19 14:54 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1860030
Environment:
Last Closed: 2020-10-19 14:54:24 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 312 0 None closed [release-4.5] Bug 1860311: Civilize logging 2020-12-21 03:42:51 UTC
Github openshift kuryr-kubernetes pull 351 0 None closed Bug 1860311: Civilize logging vol 2 2020-12-21 03:42:53 UTC
Red Hat Product Errata RHBA-2020:4228 0 None None None 2020-10-19 14:54:40 UTC

Comment 3 rlobillo 2020-08-21 12:24:35 UTC
There are still some tracebacks that are spamming the logs as indicated on Bug #1860030. 

2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     chunked=chunked)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._validate_conn(conn)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     conn.connect()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 344, in connect
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     ssl_context=context)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 354, in ssl_wrap_socket
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return context.wrap_socket(sock, server_hostname=server_hostname)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return GreenSSLSocket(sock, *a, _context=self, **kw)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     super(GreenSSLSocket, self).do_handshake)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return func(*a, **kw)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._sslobj.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._sslobj.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging OSError: [Errno 0] Error
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging 
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging During handling of the above exception, another exception occurred:
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging 
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     timeout=timeout
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     _stacktrace=sys.exc_info()[2])
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 368, in increment
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     raise six.reraise(type(error), error, _stacktrace)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 692, in reraise
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     raise value.with_traceback(tb)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     chunked=chunked)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._validate_conn(conn)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     conn.connect()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 344, in connect
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     ssl_context=context)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 354, in ssl_wrap_socket
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return context.wrap_socket(sock, server_hostname=server_hostname)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return GreenSSLSocket(sock, *a, _context=self, **kw)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     super(GreenSSLSocket, self).do_handshake)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return func(*a, **kw)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._sslobj.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._sslobj.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging urllib3.exceptions.ProtocolError: ('Connection aborted.', OSError(0, 'Error'))
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging 
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging During handling of the above exception, another exception occurred:
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging 
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._handler(event)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 67, in __call__
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._k8s.get(obj_link)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 81, in get
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     headers=header)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/api.py", line 75, in get
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return request('get', url, params=params, **kwargs)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/api.py", line 60, in request
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return session.request(method=method, url=url, **kwargs)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     resp = self.send(prep, **send_kwargs)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     r = adapter.send(request, **kwargs)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     raise ConnectionError(err, request=request)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging requests.exceptions.ConnectionError: ('Connection aborted.', OSError(0, 'Error'))
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging 
2020-08-17 10:40:21.253 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {'type': 'ADDED', 'object': {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'openshift-cluster-node-tuning-operator', 'selfLink': '/api/v1/namespaces/openshift-cluster-node-tuning-operator', 'uid': 'df907325-b35a-45be-8062-c39e86fbc05e', 'resourceVersion': '10837', 'creationTimestamp': '2020-08-17T10:13:22Z', 'annotations': {'openshift.io/node-selector': '', 'openshift.io/sa.scc.mcs': 's0:c21,c15', 'openshift.io/sa.scc.supplemental-groups': '1000450000/10000', 'openshift.io/sa.scc.uid-range': '1000450000/10000', 'openstack.org/kuryr-net-crd': 'ns-openshift-cluster-node-tuning-operator'}}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}}}: requests.exceptions.ConnectionError: ('Connection aborted.', OSError(0, 'Error'))

Comment 6 Itzik Brown 2020-10-10 23:58:46 UTC
Verified with 4.5.0-0.nightly-2020-10-09-125103

Comment 9 errata-xmlrpc 2020-10-19 14:54:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5.15 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4228


Note You need to log in before you can comment on or make changes to this bug.