Bug 1314694 - [intservice_public_109] The indices deleted by curator are still shown on Kibana UI
Summary: [intservice_public_109] The indices deleted by curator are still shown on Kib...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OKD
Classification: Red Hat
Component: Logging
Version: 3.x
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Luke Meyer
QA Contact: chunchen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-03-04 09:21 UTC by Xia Zhao
Modified: 2016-09-30 02:16 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-03-04 15:19:29 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Xia Zhao 2016-03-04 09:21:59 UTC
Problem description: 
Configure log rotation policies and make it effect, the indices deleted by curator can not be queried out from ES, but is still shown on Kibana UI with empty log entries

Version-Release number of selected component (if applicable):
oc v1.1.3-397-g326e0ed
kubernetes v1.2.0-alpha.7-703-gbc4550d
logging images are built from https://github.com/richm/origin-aggregated-logging

How reproducible:
Always

Steps to Reproduce:
1.  Local build logging component images according to https://github.com/richm/origin-aggregated-logging#defining-local-builds
2.  Deploy logging system on origin according to  https://docs.openshift.org/latest/install_config/aggregate_logging.html
3.  Wait for all logging pods running, query ES, note down the existing indices:
$ oc rsh <curator_pod>
#curator --host logging-es --use_ssl --certificate /etc/curator/keys/ca --client-cert /etc/curator/keys/cert --client-key /etc/curator/keys/key --loglevel ERROR show indices --all-indices|grep myapp
myapp-dev.2016.03.04
myapp-qe.2016.03.04
myapp-qe.2016.05.03
4.  Configure log rotation policy:
# vim /tmp/file
myapp-dev:
 delete:
   days: 1
myapp-qe:
  delete:
    weeks: 1

oc secrets new curator-config settings=/tmp/file
oc volumes dc/logging-curator --add --type=secret --secret-name=curator-config --mount-path=/etc/curator --name=curator-config --overwrite
oc scale --replicas=0 dc logging-curator
oc scale --replicas=1 dc logging-curator
5. Specify log rotation policy envs in curator dc:
    name: CURATOR_CRON_HOUR
    value: "1"
    name: CURATOR_CRON_MINUTE
    value: "1"
    name: DEFAULT_DAYS
    value: "0"
6. Adjust system time on curator pod to 2 months later, adjust the time to trigger the curator clean up cron job :
$ date -s 03/05/2016
$ date -s 01:00:00         
$ clock -w
7. Restart fluentd/es/kibana pod by scaling rc down and up
8. Wait for all logging pods become running again, querying ES to make sure log rotation policy effected:
$ oc rsh <curator_pod>
#curator --host logging-es --use_ssl --certificate /etc/curator/keys/ca --client-cert /etc/curator/keys/cert --client-key /etc/curator/keys/key --loglevel ERROR show indices --all-indices|grep myapp
myapp-qe.2016.05.03
9. Reload the Kibana UI on browser

Actual Result:
Index "myapp-dev" deleted by curator is still visible on Kibana UI, with empty log entries

Expected Result:
Index "myapp-dev" deleted by curator should not be visible on Kibana UI


Note You need to log in before you can comment on or make changes to this bug.