Bug 1517875 - RFE: Allow regular expressions in curator settings
Summary: RFE: Allow regular expressions in curator settings
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RFE
Version: 3.5.1
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.9.0
Assignee: Josef Karasek
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks: 1541948 1541952
TreeView+ depends on / blocked
 
Reported: 2017-11-27 16:09 UTC by Bahaddin
Modified: 2018-05-02 11:54 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: Allow regular expressions in curator settings Reason: Customer request Result: A special tag can be used in curator settings to specify custom regular expressions.
Clone Of:
: 1541948 (view as bug list)
Environment:
Last Closed: 2018-03-28 14:13:03 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github github.com/openshift origin-aggregated-logging pull 911 None None None 2018-01-24 10:25:32 UTC
Github https://github.com/openshift origin-aggregated-logging pull 910 None None None 2018-01-24 10:24:58 UTC
Github openshift origin-aggregated-logging pull 862 None None None 2018-01-11 11:44:11 UTC
Red Hat Product Errata RHBA-2018:0489 None None None 2018-03-28 14:13:48 UTC

Description Bahaddin 2017-11-27 16:09:53 UTC
Description of problem:

I would like to delete some indices according to the regular expressions I provide, so that I am not forced to enter individual project names into the configs.

Right now I am creating a setting yaml with a one-liner and add it to the configmap, manually. But whenever new projects are added to the cluster I will have to maintain the list.




Version-Release number of selected component (if applicable):
3.5.1

How reproducible:


Steps to Reproduce:
1. Install Cluster
2. Deploy logging with the provided playbook 
3. Add the following to the configmap logging-curator

.+nonprod.+:
  delete:
    days: 5
In order to delete the indices that belong to *nonprod projects.

Actual results:

Regex characters are escaped, as a result the indices are not deleted.

Comment 1 Rich Megginson 2017-11-27 18:27:41 UTC
ovirt/rhv would like this feature as well

Comment 2 Josef Karasek 2017-11-30 14:18:18 UTC
Bahaddin, we're currently moving to curator 5.2, which allows regexes in its configuration. See https://www.elastic.co/guide/en/elasticsearch/client/curator/5.0/filtertype_pattern.html#_regex

Let me know if that helps you. Thanks

Comment 3 Rich Megginson 2017-11-30 14:21:34 UTC
(In reply to Josef Karasek from comment #2)
> Bahaddin, we're currently moving to curator 5.2, which allows regexes in its
> configuration. See
> https://www.elastic.co/guide/en/elasticsearch/client/curator/5.0/
> filtertype_pattern.html#_regex
> 
> Let me know if that helps you. Thanks

Josef, is there anything we can do in current version of curator?

Comment 4 Bahaddin 2017-11-30 19:51:14 UTC
(In reply to Josef Karasek from comment #2)
> Bahaddin, we're currently moving to curator 5.2, which allows regexes in its
> configuration. See
> https://www.elastic.co/guide/en/elasticsearch/client/curator/5.0/
> filtertype_pattern.html#_regex
> 
> Let me know if that helps you. Thanks

Hi Josef,

which OCP version will be using the curator 5.2?

Thanks,
Bahaddin

Comment 5 Josef Karasek 2017-12-01 09:36:54 UTC
We're hoping to land this upgrade in early 2018.
I understand that this might not satisfied your needs.
I'll look into extending the current config file format to accept regular expressions as well.

Comment 9 Anping Li 2018-02-27 05:57:38 UTC
The latest image logging-curator/images/v3.9.0-0.53.0.0 won't include the fix. Waiting the new image.

Comment 11 Anping Li 2018-03-09 15:03:34 UTC
@josef, The curator report error, Could you Check if the configmap are correct?
[root@ip-172-18-5-104 bin]# oc get configmap logging-curator -o yaml
apiVersion: v1
data:
  config.yaml: |
    myapp-dev:
      delete:
        days: 1

    myapp-qe:
      delete:
        weeks: 1

    .operations:
      delete:
        weeks: 2

    .defaults:
      delete:
        days: 31
    runhour: 0
    runminute: 0
    timezone: America/New_York

    .regex:
    - pattern: '^project\..+\-dev\..*$'
      delete:
        days: 15
    - pattern: '^project\..+\-test\..*$'
      delete:
        days: 30
kind: ConfigMap


[root@ip-172-18-5-104 bin]# oc logs logging-curator-1-6sptm
The project name must match this regex: [^[a-z0-9]([-a-z0-9]*[a-z0-9])?$] This does not match: [.regex]

Comment 12 Josef Karasek 2018-03-14 09:19:36 UTC
The test fails because none of the latest builds have the latest code. I'm looking into it.

The indentation of the config should be:
config.yaml: |
  myapp-dev:
    delete:
      days: 1

  myapp-qe:
    delete:
      weeks: 1

  .operations:
    delete:
      weeks: 2

  .defaults:
    delete:
      days: 31
    runhour: 0
    runminute: 0
    timezone: America/New_York

  .regex:
    - pattern: '^project\..+\-dev\..*$'
      delete:
        days: 15
    - pattern: '^project\..+\-test\..*$'
      delete:
        days: 30

You can find more info here https://github.com/openshift/origin-aggregated-logging/blob/release-3.9/curator/README.md

Comment 13 Jeff Cantrill 2018-03-14 13:02:46 UTC
Moving back to RFE as this was a change enhancement.  Updated dist-git repos to include latest files due to bad merge.  Requires new build to validate.

Comment 14 Anping Li 2018-03-19 09:08:03 UTC
The curator works as expected with openshift3/logging-curator/images/v3.9.11-1

Comment 17 errata-xmlrpc 2018-03-28 14:13:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489


Note You need to log in before you can comment on or make changes to this bug.