Problem description: Configure log rotation policies and make it effect, the indices deleted by curator can not be queried out from ES, but is still shown on Kibana UI with empty log entries Version-Release number of selected component (if applicable): oc v1.1.3-397-g326e0ed kubernetes v1.2.0-alpha.7-703-gbc4550d logging images are built from https://github.com/richm/origin-aggregated-logging How reproducible: Always Steps to Reproduce: 1. Local build logging component images according to https://github.com/richm/origin-aggregated-logging#defining-local-builds 2. Deploy logging system on origin according to https://docs.openshift.org/latest/install_config/aggregate_logging.html 3. Wait for all logging pods running, query ES, note down the existing indices: $ oc rsh <curator_pod> #curator --host logging-es --use_ssl --certificate /etc/curator/keys/ca --client-cert /etc/curator/keys/cert --client-key /etc/curator/keys/key --loglevel ERROR show indices --all-indices|grep myapp myapp-dev.2016.03.04 myapp-qe.2016.03.04 myapp-qe.2016.05.03 4. Configure log rotation policy: # vim /tmp/file myapp-dev: delete: days: 1 myapp-qe: delete: weeks: 1 oc secrets new curator-config settings=/tmp/file oc volumes dc/logging-curator --add --type=secret --secret-name=curator-config --mount-path=/etc/curator --name=curator-config --overwrite oc scale --replicas=0 dc logging-curator oc scale --replicas=1 dc logging-curator 5. Specify log rotation policy envs in curator dc: name: CURATOR_CRON_HOUR value: "1" name: CURATOR_CRON_MINUTE value: "1" name: DEFAULT_DAYS value: "0" 6. Adjust system time on curator pod to 2 months later, adjust the time to trigger the curator clean up cron job : $ date -s 03/05/2016 $ date -s 01:00:00 $ clock -w 7. Restart fluentd/es/kibana pod by scaling rc down and up 8. Wait for all logging pods become running again, querying ES to make sure log rotation policy effected: $ oc rsh <curator_pod> #curator --host logging-es --use_ssl --certificate /etc/curator/keys/ca --client-cert /etc/curator/keys/cert --client-key /etc/curator/keys/key --loglevel ERROR show indices --all-indices|grep myapp myapp-qe.2016.05.03 9. Reload the Kibana UI on browser Actual Result: Index "myapp-dev" deleted by curator is still visible on Kibana UI, with empty log entries Expected Result: Index "myapp-dev" deleted by curator should not be visible on Kibana UI