Bug 1369863

Summary: Curator should automatically clean up deleted projects from elasticsearch
Product: OpenShift Container Platform Reporter: Wesley Hearn <whearn>
Component: LoggingAssignee: Luke Meyer <lmeyer>
Status: CLOSED WONTFIX QA Contact: Xia Zhao <xiazhao>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.2.1CC: aos-bugs, ewolinet, jcantril, rmeggins
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-12 15:25:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1303130    

Description Wesley Hearn 2016-08-24 14:45:17 UTC 
Description of problem:
Curator should delete deleted namespaces from elasticsearch on a nightly basis.
Currently our monitoring goes through and creates an app every X mins, makes sure it is up and running then deletes the NS. Everytime this runs it creates a new ES index for the project(fix for CVE-2016-2149). How ever this easily overloads elasticsearch after a week or so running.

Version-Release number of selected component (if applicable):
3.2.1

How reproducible:
Always

Steps to Reproduce:
1. Setup 3.2.1 OCP cluster with logging
2. Have a cron job that creates and deletes a project and app every 5 mins
3. Watch ElasticSearch indices

Actual results:
The indices count builds up over time and will only get cleaned up after X days(as defined in the curator)

Expected results:
The curator should clean up deleted projects faster then it prune application logs.

Additional info:
Comment 1 Rich Megginson 2016-08-24 15:38:23 UTC 
> 2. Have a cron job that creates and deletes a project and app every 5 mins

Same project name?  If so, you can configure curator to delete that project name every day.  That should delete all 288 indices every day.

Another way to solve this particular problem - we could configure fluentd to ignore certain project name patterns.  When using OSE 3.3:

create a configmap for fluentd that creates a file in the fluentd pod called /etc/fluent/configs.d/openshift/filter-pre-omit-monitor-project.conf

This file looks like this:

<match kubernetes.journal.container.myprojectname** kubernetes.var.log.containers.**myprojectname**>
  @type null
</match>

That is, if using journald for the docker log-driver and the tag matches the project name, or using json-file, just throw away the record.  That way, the data never reaches elasticsearch.
Comment 2 Jeff Cantrill 2016-08-25 14:02:37 UTC 
Lowering priority as we have a recommendation for resolution: https://bugzilla.redhat.com/show_bug.cgi?id=1369863#c1
Comment 4 Luke Meyer 2016-10-12 15:25:38 UTC 
The suggestions are to limit the lifetime of a test project that's recreated repeatedly. I don't think we actually want, in general, to delete indices for deleted projects on a schedule different from persisting ones. One of the benefits of aggregated logging is the ability to look at deleted logs.

Closing this bug unless someone objects.