We are trying to revert the curator image to use the curator package version 5.2 - but CI is currently broken - see https://github.com/openshift/release/issues/2106 and https://github.com/openshift/aos-cd-jobs/pull/1526

https://github.com/openshift/origin-aggregated-logging/pull/1446

ART needs to sync and rebuild the curator5 image

I am still getting the same curator python stacktrace, exact same message using the the reverted 5.2.0 version in ocp 3.11.43.

We discovered one more problem - we need to use the previous version of python-elasticsearch.  We have tagged python-elasticsearch-5.4.0-1.el7 into rhaos-3.11-rhel-7-candidate and rhaos-4.0-rhel-7-candidate and are waiting for ART to rebuild the composes with these packages and rebuild the curator images.

For anyone else hitting this, the v3.11.23 image works as a temporary workaround.

Verified in ose-logging-curator5:v3.11.50

(In reply to Matthew Barnes from comment #9)
> For anyone else hitting this, the v3.11.23 image works as a temporary
> workaround.

It seems that v3.11.23 is not longer available, am I right?

(In reply to Qiaoling Tang from comment #11)
> Verified in ose-logging-curator5:v3.11.50

Where did you find .50? I guess latest tag right now points to v3.11.43 [1], which is the one I have on a fresh 3.11 installation, where the issue is still present.

[1] https://access.redhat.com/containers/?tab=tags#/registry.access.redhat.com/openshift3/ose-logging-curator5

(In reply to Serena Cortopassi from comment #12)
> (In reply to Matthew Barnes from comment #9)
> > For anyone else hitting this, the v3.11.23 image works as a temporary
> > workaround.
> 
> It seems that v3.11.23 is not longer available, am I right?

If you are using openshift Enterprise: As far as I know it never was. released versions were 3.11.16 and 3.11.43. 

As a workaround I have created a onetime Job in my setting that will clean out the logs - My disks were running full and I needed spontaneous action. The Job uses the curator image from openshift origin (your cluster needs to be able to download from docker hub). Yaml-File for the ontime job is below. I guess the same image would also work with the standard CronJob setup by openshift enterprise, but I have not tested that.

-----snip-----
apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: null
  name: logging-curator-onetime
spec:
  backoffLimit: 0
  template:
    metadata:
      creationTimestamp: null
      labels:
        component: curator
        logging-infra: curator
        provider: openshift
      name: logging-curator-onetime
    spec:
      containers:
      - env:
        - name: K8S_HOST_URL
          value: https://kubernetes.default.svc.cluster.local
        - name: ES_HOST
          value: logging-es
        - name: ES_PORT
          value: "9200"
        - name: ES_CLIENT_CERT
          value: /etc/curator/keys/cert
        - name: ES_CLIENT_KEY
          value: /etc/curator/keys/key
        - name: ES_CA
          value: /etc/curator/keys/ca
        - name: CURATOR_DEFAULT_DAYS
          value: "7"
        - name: CURATOR_SCRIPT_LOG_LEVEL
          value: DEBUG
        - name: CURATOR_LOG_LEVEL
          value: ERROR
        - name: CURATOR_TIMEOUT
          value: "300"
        image: openshift/origin-logging-curator5:v3.11
        imagePullPolicy: IfNotPresent
        name: curator
        resources:
          limits:
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 256Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/curator/keys
          name: certs
          readOnly: true
        - mountPath: /etc/curator/settings
          name: config
          readOnly: true
      dnsPolicy: ClusterFirst
      nodeSelector:
        node-role.kubernetes.io/infra: "true"
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: aggregated-logging-curator
      serviceAccountName: aggregated-logging-curator
      terminationGracePeriodSeconds: 30
      volumes:
      - name: certs
        secret:
          defaultMode: 420
          secretName: logging-curator
      - configMap:
          defaultMode: 420
          name: logging-curator
        name: config

The tentative release date for the updated image is Dec. 11

*** Bug 1657560 has been marked as a duplicate of this bug. ***

AFAICT the fix is in openshift3/ose-logging-curator5:v3.11.54-1

>docker run -it openshift3/ose-logging-curator5:v3.11.54-1 bash
...
bash-4.2$ curator --help
Usage: curator [OPTIONS] ACTION_FILE

  Curator for Elasticsearch indices.

  See http://elastic.co/guide/en/elasticsearch/client/curator/current

Options:
  --config PATH  Path to configuration file. Default: ~/.curator/curator.yml
  --dry-run      Do not perform any changes.
  --version      Show the version and exit.
  --help         Show this message and exit.


If I use a broken version I get an error message:

>docker run -it openshift3/ose-logging-curator5:v3.11.51-3 bash
...
bash-4.2$ curator --help
Traceback (most recent call last):
  File "/usr/bin/curator", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 3007, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 728, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 626, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: elasticsearch>=5.4.0,<6.0.0

I've looked at the list of python packages from the last known good version of curator v3.11.23-1 and compared them to v3.11.54-1.  The only difference is the python-setuptools package, which seems to have reverted from python-setuptools-17.1.1-4.el7.noarch in v3.11.23-1 to python-setuptools-0.9.8-7.el7.noarch in 3.11.54?

@Qiaoling - can you verify ose-logging-curator5:v3.11.54 - we really need to get this fix into the errata that is shipping tomorrow.

It is  ose-logging-curator5:v3.11.51 in https://errata.devel.redhat.com/advisory/38178. But the fix is ose-logging-curator5:v3.11.54. So the bug is not fix in advisory/38178. Move to modified.

Verified in openshift3/ose-logging-curator5/images/v3.11.59-2.

Steps and results are the same as comment 18

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0024