Bug 1524644 - Curator configuration produces wrong regular expressions
Summary: Curator configuration produces wrong regular expressions
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.7.z
Assignee: Josef Karasek
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-12-11 19:23 UTC by Josef Karasek
Modified: 2018-04-05 09:33 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Script that parses user supplied configuration produced wrong regular expression. Consequence: No impact on users - the non-escaped period (.) in the regular expression matched the period (.) in the project name as 'any character'. Fix: Properly escape characters that should not be expanded as special characters in regular expressions. Result: Example regular expression produced after applying this patch: '^\.operations\..*$' instead of '.operations.'
Clone Of:
Environment:
Last Closed: 2018-04-05 09:33:10 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 878 None None None 2018-01-11 12:20:18 UTC
Red Hat Product Errata RHBA-2018:0636 None None None 2018-04-05 09:33:53 UTC

Description Josef Karasek 2017-12-11 19:23:20 UTC
Description of problem:
User supplied configuration is translated into unintended regular expressions.
Currently things work by sheer luck


Version-Release number of selected component (if applicable):
origin aggregated logging 3.7.0


How reproducible:
always, just run curator with a valid configuration


Steps to Reproduce:
1. supply some valid config to curator
2. check the curator commands that run_cron.py generated
3.

Actual results:
/usr/bin/curator \
    ... # other config \
    --exclude .searchguard* \
    --exclude .kibana* \
    --exclude '\.operations\.*' \
    --exclude 'myapp\-qe\.*' \
    --exclude 'proj1\.*' \
    --exclude 'myapp\-dev\.*'

Expected results:
/usr/bin/curator \
    ... # other config \
    --exclude '^\.searchguard\..*$' 
    --exclude '^\.kibana\..*$' 
    --exclude '^\.operations\..*$' 
    --exclude '^project\.myapp\-qe\..*$' 
    --exclude '^project\.proj1\..*$' 
    --exclude '^project\.myapp\-dev\..*$'

Additional info:
Easy way how to debug/test this:
$ python
>>> import re
>>> list(filter(lambda x: not re.search(r'^project\.myapp\-qe\..*$', x), ['project.myapp-qe.', 'abc']))

graphical representation of regexes: https://www.debuggex.com/

Comment 1 openshift-github-bot 2018-01-02 18:33:46 UTC
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/9476cabda226b2df0e2a99149dea3a238ab01c8b
bug 1524644. Curator configuration produces wrong regular expressions

For example `^.operations.` instead of `^\.operations\..*$`
Both regexes match, but the first one only accidentaly. Example index name:
`.operations.1827ac8a-ef8f-11e7-92de-0ebdd5807cb2.2018.01.02`

Curator code: https://github.com/elastic/curator/blob/3.5/curator/api/filter.py#L90-L138

https://github.com/openshift/origin-aggregated-logging/commit/5ae3d1fbda7f4e0686a96a000f56dbb61a0d6226
Merge pull request #876 from josefkarasek/bz-1524644

Automatic merge from submit-queue.

bug 1524644. Curator configuration produces wrong regular expressions

For example `^.operations.` instead of `^\.operations\..*$`
Both regexes match, but the first one only accidentaly. Example index name:
`.operations.1827ac8a-ef8f-11e7-92de-0ebdd5807cb2.2018.01.02`

Curator code: https://github.com/elastic/curator/blob/3.5/curator/api/filter.py#L90-L138

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1524644

Comment 2 Anping Li 2018-01-15 08:33:00 UTC
Verified and pass on openshift3/logging-curator/images/v3.7.23-2
1. .searchguard and .kibana index are excluded.
2. index are deleted after 30 days
3. The index can be delete according the the values in configmap logging-curator

Comment 3 Josef Karasek 2018-01-15 10:51:38 UTC
v3.7.23-2 does not have the changes. Will have to wait for a new build.

Comment 5 Anping Li 2018-02-25 04:12:57 UTC
@josef, The change [1] is still not in the latest image openshift3/logging-curator/images/v3.7.31-1.  Could you confirm if the code are merged to v3.7.

[1]https://github.com/openshift/origin-aggregated-logging/commit/5ae3d1fbda7f4e0686a96a000f56dbb61a0d6226

Comment 6 Josef Karasek 2018-02-26 06:57:45 UTC
I issued one more change to the code, it was merged in the master and 3.7 branch on 21st february.

https://github.com/openshift/origin-aggregated-logging/pull/958
https://github.com/openshift/origin-aggregated-logging/pull/963

Comment 7 Jeff Cantrill 2018-03-12 16:59:16 UTC
Should be available in: v3.7.38-1 or later

Comment 9 Anping Li 2018-03-30 01:50:32 UTC
Works well with 3.7.40

Comment 13 errata-xmlrpc 2018-04-05 09:33:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0636


Note You need to log in before you can comment on or make changes to this bug.