Bug 1468520 - log deletion only effect to one index among those with duplicated policies
log deletion only effect to one index among those with duplicated policies
Status: CLOSED NOTABUG
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging (Show other bugs)
3.6.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Jeff Cantrill
Xia Zhao
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-07 06:16 EDT by Xia Zhao
Modified: 2017-07-09 21:46 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-07-07 14:22:59 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Xia Zhao 2017-07-07 06:16:16 EDT
Description of problem:
log deletion only effect to one index (the first one specified in curator-config.yaml) among those with duplicated policies.
Similar old bug: https://bugzilla.redhat.com/show_bug.cgi?id=1352486

Version-Release number of selected component (if applicable):
Image:
logging-curator            v3.6                52f85e17fd38        6 hours ago         221.5 MB

# openshift version
openshift v3.6.136
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

How reproducible:
always

Steps to Reproduce:
1. Note down the current indices for your interest in openshift:

$ oc exec $curator-pod -- curator --host logging-es --use_ssl --certificate /etc/curator/keys/ca --client-cert /etc/curator/keys/cert --client-key /etc/curator/keys/key --loglevel ERROR show indices --all-indices

project.day1-dup.2646499a-ca50-457d-9cc4-5179bda279c7.2016.2016.09.01
project.day1-dup.4f47d263-c25e-4def-99cb-53963edab8a4.2016.2017.7.7
project.day1-tri.4dfc53c1-9176-4d89-be0b-f2b701a3eee2.2016.2017.7.7
project.day1-tri.62edf619-4948-4cbf-bbf4-c69db628832d.2016.2016.08.29
project.day1.3464b2d1-f4c6-4900-813a-8c17fa0914e5.2016.2017.7.7
project.day1.fd26339e-87fc-49bd-bc19-860d43eac070.2016.2016.08.29

2. Configure curator settings like this (by editting the configmap) :

$ oc edit configmap logging-curator -o yaml
...
    day1:
      delete:
        days: 1
    day-dup:
      delete:
        days: 1
    day-tri:
      delete:
        days: 1
...

3. Scale down curator pod:
oc scale rc logging-curator-1 --replicas=0
4. Scale up curator pod:
oc scale rc logging-curator-1 --replicas=1

5. Check the index deletion result:

$  oc exec $curator-pod -- curator --host logging-es --use_ssl  --certificate /etc/curator/keys/ca --client-cert /etc/curator/keys/cert  --client-key /etc/curator/keys/key --loglevel ERROR show indices  --all-indices

project.day1-dup.2646499a-ca50-457d-9cc4-5179bda279c7.2016.2016.09.01
project.day1-dup.4f47d263-c25e-4def-99cb-53963edab8a4.2016.2017.7.7
project.day1-tri.4dfc53c1-9176-4d89-be0b-f2b701a3eee2.2016.2017.7.7
project.day1-tri.62edf619-4948-4cbf-bbf4-c69db628832d.2016.2016.08.29
project.day1.3464b2d1-f4c6-4900-813a-8c17fa0914e5.2016.2017.7.7

Actual results:
old logs more than 1 day for project day-dup and day-tri are not deleted, only those for project day1 deleted

Expected results:
old logs more than 1 day for project day-dup and day-tri should also be deleted

Additional info:
Comment 1 Rich Megginson 2017-07-07 14:22:59 EDT
The indices all start with the prefix "project.day1-" but your curator config project names _do not_:

    day-dup:
      delete:
        days: 1
    day-tri:
      delete:
        days: 1

It should work if you either change your index names to be "project.day-dup" and "project.day-tri", or change your curator config like this:

    day1-dup:
      delete:
        days: 1
    day1-tri:
      delete:
        days: 1

In addition - curator works with the index naming scheme used by fluentd/elasticsearch:

project.name.uuid.YYYY.MM.DD

Where MM and DD are the _2_ digit month and day e.g. 07 instead of 7.  Using any other index naming convention may work but is not supported.
Comment 2 Xia Zhao 2017-07-09 21:46:17 EDT
(In reply to Rich Megginson from comment #1)
> The indices all start with the prefix "project.day1-" but your curator
> config project names _do not_:
Woo, my appologize for the mistake, thanks for pointing it out, Rich!

> Where MM and DD are the _2_ digit month and day e.g. 07 instead of 7.  Using
> any other index naming convention may work but is not supported.

Thanks for the info, I'll modify my script to make the timestamps in _2_ digit format when addming messages into curator.

Note You need to log in before you can comment on or make changes to this bug.