Bug 1314694

Summary: [intservice_public_109] The indices deleted by curator are still shown on Kibana UI
Product: OKD Reporter: Xia Zhao <xiazhao>
Component: LoggingAssignee: Luke Meyer <lmeyer>
Status: CLOSED NOTABUG QA Contact: chunchen <chunchen>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.xCC: aos-bugs, ewolinet, rmeggins, wsun
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-04 15:19:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Xia Zhao 2016-03-04 09:21:59 UTC 
Problem description: 
Configure log rotation policies and make it effect, the indices deleted by curator can not be queried out from ES, but is still shown on Kibana UI with empty log entries

Version-Release number of selected component (if applicable):
oc v1.1.3-397-g326e0ed
kubernetes v1.2.0-alpha.7-703-gbc4550d
logging images are built from https://github.com/richm/origin-aggregated-logging

How reproducible:
Always

Steps to Reproduce:
1.  Local build logging component images according to https://github.com/richm/origin-aggregated-logging#defining-local-builds
2.  Deploy logging system on origin according to  https://docs.openshift.org/latest/install_config/aggregate_logging.html
3.  Wait for all logging pods running, query ES, note down the existing indices:
$ oc rsh <curator_pod>
#curator --host logging-es --use_ssl --certificate /etc/curator/keys/ca --client-cert /etc/curator/keys/cert --client-key /etc/curator/keys/key --loglevel ERROR show indices --all-indices|grep myapp
myapp-dev.2016.03.04
myapp-qe.2016.03.04
myapp-qe.2016.05.03
4.  Configure log rotation policy:
# vim /tmp/file
myapp-dev:
 delete:
   days: 1
myapp-qe:
  delete:
    weeks: 1

oc secrets new curator-config settings=/tmp/file
oc volumes dc/logging-curator --add --type=secret --secret-name=curator-config --mount-path=/etc/curator --name=curator-config --overwrite
oc scale --replicas=0 dc logging-curator
oc scale --replicas=1 dc logging-curator
5. Specify log rotation policy envs in curator dc:
    name: CURATOR_CRON_HOUR
    value: "1"
    name: CURATOR_CRON_MINUTE
    value: "1"
    name: DEFAULT_DAYS
    value: "0"
6. Adjust system time on curator pod to 2 months later, adjust the time to trigger the curator clean up cron job :
$ date -s 03/05/2016
$ date -s 01:00:00         
$ clock -w
7. Restart fluentd/es/kibana pod by scaling rc down and up
8. Wait for all logging pods become running again, querying ES to make sure log rotation policy effected:
$ oc rsh <curator_pod>
#curator --host logging-es --use_ssl --certificate /etc/curator/keys/ca --client-cert /etc/curator/keys/cert --client-key /etc/curator/keys/key --loglevel ERROR show indices --all-indices|grep myapp
myapp-qe.2016.05.03
9. Reload the Kibana UI on browser

Actual Result:
Index "myapp-dev" deleted by curator is still visible on Kibana UI, with empty log entries

Expected Result:
Index "myapp-dev" deleted by curator should not be visible on Kibana UI