Hide Forgot
Description of problem: Curator should delete deleted namespaces from elasticsearch on a nightly basis. Currently our monitoring goes through and creates an app every X mins, makes sure it is up and running then deletes the NS. Everytime this runs it creates a new ES index for the project(fix for CVE-2016-2149). How ever this easily overloads elasticsearch after a week or so running. Version-Release number of selected component (if applicable): 3.2.1 How reproducible: Always Steps to Reproduce: 1. Setup 3.2.1 OCP cluster with logging 2. Have a cron job that creates and deletes a project and app every 5 mins 3. Watch ElasticSearch indices Actual results: The indices count builds up over time and will only get cleaned up after X days(as defined in the curator) Expected results: The curator should clean up deleted projects faster then it prune application logs. Additional info:
> 2. Have a cron job that creates and deletes a project and app every 5 mins Same project name? If so, you can configure curator to delete that project name every day. That should delete all 288 indices every day. Another way to solve this particular problem - we could configure fluentd to ignore certain project name patterns. When using OSE 3.3: create a configmap for fluentd that creates a file in the fluentd pod called /etc/fluent/configs.d/openshift/filter-pre-omit-monitor-project.conf This file looks like this: <match kubernetes.journal.container.myprojectname** kubernetes.var.log.containers.**myprojectname**> @type null </match> That is, if using journald for the docker log-driver and the tag matches the project name, or using json-file, just throw away the record. That way, the data never reaches elasticsearch.
Lowering priority as we have a recommendation for resolution: https://bugzilla.redhat.com/show_bug.cgi?id=1369863#c1
The suggestions are to limit the lifetime of a test project that's recreated repeatedly. I don't think we actually want, in general, to delete indices for deleted projects on a schedule different from persisting ones. One of the benefits of aggregated logging is the ability to look at deleted logs. Closing this bug unless someone objects.