Bug 1465713

Summary: Traceback info in curator pod's log
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: LoggingAssignee: Rich Megginson <rmeggins>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.6.0CC: aos-bugs, jcantril, jeder, jmontleo, pportant, rmeggins, xiazhao
Target Milestone: ---   
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
URL: aos-scalability-36
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-28 21:58:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
curator pods' log none

Description Junqi Zhao 2017-06-28 01:35:29 UTC
Created attachment 1292538 [details]
curator pods' log

Description of problem:
Traceback info in curator pod's log
# oc logs logging-curator-1-12sq7
curator running [1] jobs
Traceback (most recent call last):
  File "/usr/bin/curator", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3084, in <module>
    @_call_aside
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3070, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3097, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 653, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 666, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 839, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'urllib3<2.0,>=1.8' distribution was not found and is required by elasticsearch

Version-Release number of selected component (if applicable):
Images from brew registry
logging-elasticsearch      v3.6                19ad6f8e4738        29 minutes ago      404.2 MB
logging-auth-proxy         v3.6                d94bddb3dcba        8 hours ago         214.8 MB
logging-kibana             v3.6                4eabc3acd717        21 hours ago        342.4 MB
logging-fluentd            v3.6                08e8a59602fe        21 hours ago        232.5 MB
logging-curator            v3.6                a0148dd96b8d        2 weeks ago         221.5 MB

How reproducible:
Always

Steps to Reproduce:
1. Deploy logging via ansible
2. Check curator pod's log
3.

Actual results:
Traceback info in curator pod's log

Expected results:
Should not be exception info in pod's log

Additional info:

Comment 1 Rich Megginson 2017-06-28 02:19:08 UTC
I just made a new build to check which packages are available:
2017-06-28 01:55:45,841 - atomic_reactor.plugins.orchestrate_build - INFO - x86_64 - 2017-06-28 01:55:45,826 - atomic_reactor.util - DEBUG - --> Processing Dependency: python-urllib3 for package: elastic-curator-3.5.0-2.el7.noarch
2017-06-28 01:55:45,898 - atomic_reactor.plugins.orchestrate_build - INFO - x86_64 - 2017-06-28 01:55:45,846 - atomic_reactor.util - DEBUG - ---> Package python2-urllib3.noarch 0:1.21-1.el7 will be installed
2017-06-28 01:55:46,350 - atomic_reactor.plugins.orchestrate_build - INFO - x86_64 - 2017-06-28 01:55:46,234 - atomic_reactor.util - DEBUG - python2-urllib3    noarch 1.21-1.el7          rhel-7-server-ose-3.6-rpms 172 k

This looks correct: python2-urllib3    noarch 1.21-1.el7

I'm not sure what could have happened.

Please try this image:
koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=569038
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-curator:rhaos-3.6-rhel-7-docker-candidate-76378-20170628015440
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-curator:3.6.99
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-curator:3.6.99-3
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-curator:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-curator:v3.6

Comment 2 Junqi Zhao 2017-06-28 04:23:46 UTC
curator pod's log still has Traceback info, same error with Comment 0

# python --version
Python 2.7.5

# docker images | grep logging
brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-curator         3.6.99-3            757cf4948069        2 hours ago         221.5 MB
brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-curator         v3.6                757cf4948069        2 hours ago         221.5 MB

Comment 3 Jason Montleon 2017-06-30 17:47:25 UTC
I think the package is reporting the wrong version.

>>> import urllib3
>>> print urllib3.__version__
dev


>>> import pkg_resources
>>> pkg_resources.get_distribution("urllib3").version
'dev'

instead of 1.21. Looking at the upstream it looks like it's the correct source commit, but the wrong branch (master vs. release). I should have ann updated package shortly.

Comment 7 Junqi Zhao 2017-07-03 23:49:33 UTC
Tested, no Traceback info in curator pod's log now.

# oc logs ${curator_pod}
curator running [1] jobs
No indices matched provided args: {'regex': None, 'index': (), 'suffix': None, 'newer_than': None, 'closed_only': False, 'prefix': None, 'time_unit': 'days', 'timestring': '%Y.%m.%d', 'exclude': ('.searchguard*', '.kibana*'), 'older_than': 30, 'all_indices': False}
curator run finish

# openshift version
openshift v3.6.132
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

Curator images from brew registry
logging-curator         v3.6                b609245a72f9        17 hours ago        221.5 MB
logging-curator         v3.6.132-1          b609245a72f9        17 hours ago        221.5 MB

Comment 11 errata-xmlrpc 2017-11-28 21:58:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188