Bug 1860311 - kuryr-controller logs are awfully unreadable
Summary: kuryr-controller logs are awfully unreadable
Keywords:
Status: ASSIGNED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.5.z
Assignee: Michał Dulko
QA Contact: GenadiC
URL:
Whiteboard:
Depends On: 1860030
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-24 09:51 UTC by Luis Tomas Bolivar
Modified: 2020-09-24 06:45 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1860030
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 312 None closed [release-4.5] Bug 1860311: Civilize logging 2020-09-21 06:37:03 UTC
Github openshift kuryr-kubernetes pull 351 None open Bug 1860311: Civilize logging vol 2 2020-09-23 15:31:54 UTC

Comment 3 rlobillo 2020-08-21 12:24:35 UTC
There are still some tracebacks that are spamming the logs as indicated on Bug #1860030. 

2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     chunked=chunked)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._validate_conn(conn)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     conn.connect()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 344, in connect
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     ssl_context=context)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 354, in ssl_wrap_socket
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return context.wrap_socket(sock, server_hostname=server_hostname)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return GreenSSLSocket(sock, *a, _context=self, **kw)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     super(GreenSSLSocket, self).do_handshake)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return func(*a, **kw)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._sslobj.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._sslobj.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging OSError: [Errno 0] Error
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging 
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging During handling of the above exception, another exception occurred:
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging 
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     timeout=timeout
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     _stacktrace=sys.exc_info()[2])
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 368, in increment
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     raise six.reraise(type(error), error, _stacktrace)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 692, in reraise
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     raise value.with_traceback(tb)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     chunked=chunked)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._validate_conn(conn)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     conn.connect()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 344, in connect
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     ssl_context=context)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 354, in ssl_wrap_socket
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return context.wrap_socket(sock, server_hostname=server_hostname)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return GreenSSLSocket(sock, *a, _context=self, **kw)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     super(GreenSSLSocket, self).do_handshake)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return func(*a, **kw)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._sslobj.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._sslobj.do_handshake()
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging urllib3.exceptions.ProtocolError: ('Connection aborted.', OSError(0, 'Error'))
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging 
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging During handling of the above exception, another exception occurred:
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging 
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._handler(event)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 67, in __call__
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     self._k8s.get(obj_link)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 81, in get
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     headers=header)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/api.py", line 75, in get
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return request('get', url, params=params, **kwargs)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/api.py", line 60, in request
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     return session.request(method=method, url=url, **kwargs)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     resp = self.send(prep, **send_kwargs)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     r = adapter.send(request, **kwargs)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging     raise ConnectionError(err, request=request)
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging requests.exceptions.ConnectionError: ('Connection aborted.', OSError(0, 'Error'))
2020-08-17 10:40:21.245 1 ERROR kuryr_kubernetes.handlers.logging 
2020-08-17 10:40:21.253 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {'type': 'ADDED', 'object': {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'openshift-cluster-node-tuning-operator', 'selfLink': '/api/v1/namespaces/openshift-cluster-node-tuning-operator', 'uid': 'df907325-b35a-45be-8062-c39e86fbc05e', 'resourceVersion': '10837', 'creationTimestamp': '2020-08-17T10:13:22Z', 'annotations': {'openshift.io/node-selector': '', 'openshift.io/sa.scc.mcs': 's0:c21,c15', 'openshift.io/sa.scc.supplemental-groups': '1000450000/10000', 'openshift.io/sa.scc.uid-range': '1000450000/10000', 'openstack.org/kuryr-net-crd': 'ns-openshift-cluster-node-tuning-operator'}}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}}}: requests.exceptions.ConnectionError: ('Connection aborted.', OSError(0, 'Error'))


Note You need to log in before you can comment on or make changes to this bug.