Bug 1314694 - [intservice_public_109] The indices deleted by curator are still shown on Kibana UI
[intservice_public_109] The indices deleted by curator are still shown on Kib...
Status: CLOSED NOTABUG
Product: OpenShift Origin
Classification: Red Hat
Component: Logging (Show other bugs)
3.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Luke Meyer
chunchen
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-04 04:21 EST by Xia Zhao
Modified: 2016-09-29 22:16 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-04 10:19:29 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Xia Zhao 2016-03-04 04:21:59 EST
Problem description: 
Configure log rotation policies and make it effect, the indices deleted by curator can not be queried out from ES, but is still shown on Kibana UI with empty log entries

Version-Release number of selected component (if applicable):
oc v1.1.3-397-g326e0ed
kubernetes v1.2.0-alpha.7-703-gbc4550d
logging images are built from https://github.com/richm/origin-aggregated-logging

How reproducible:
Always

Steps to Reproduce:
1.  Local build logging component images according to https://github.com/richm/origin-aggregated-logging#defining-local-builds
2.  Deploy logging system on origin according to  https://docs.openshift.org/latest/install_config/aggregate_logging.html
3.  Wait for all logging pods running, query ES, note down the existing indices:
$ oc rsh <curator_pod>
#curator --host logging-es --use_ssl --certificate /etc/curator/keys/ca --client-cert /etc/curator/keys/cert --client-key /etc/curator/keys/key --loglevel ERROR show indices --all-indices|grep myapp
myapp-dev.2016.03.04
myapp-qe.2016.03.04
myapp-qe.2016.05.03
4.  Configure log rotation policy:
# vim /tmp/file
myapp-dev:
 delete:
   days: 1
myapp-qe:
  delete:
    weeks: 1

oc secrets new curator-config settings=/tmp/file
oc volumes dc/logging-curator --add --type=secret --secret-name=curator-config --mount-path=/etc/curator --name=curator-config --overwrite
oc scale --replicas=0 dc logging-curator
oc scale --replicas=1 dc logging-curator
5. Specify log rotation policy envs in curator dc:
    name: CURATOR_CRON_HOUR
    value: "1"
    name: CURATOR_CRON_MINUTE
    value: "1"
    name: DEFAULT_DAYS
    value: "0"
6. Adjust system time on curator pod to 2 months later, adjust the time to trigger the curator clean up cron job :
$ date -s 03/05/2016
$ date -s 01:00:00         
$ clock -w
7. Restart fluentd/es/kibana pod by scaling rc down and up
8. Wait for all logging pods become running again, querying ES to make sure log rotation policy effected:
$ oc rsh <curator_pod>
#curator --host logging-es --use_ssl --certificate /etc/curator/keys/ca --client-cert /etc/curator/keys/cert --client-key /etc/curator/keys/key --loglevel ERROR show indices --all-indices|grep myapp
myapp-qe.2016.05.03
9. Reload the Kibana UI on browser

Actual Result:
Index "myapp-dev" deleted by curator is still visible on Kibana UI, with empty log entries

Expected Result:
Index "myapp-dev" deleted by curator should not be visible on Kibana UI

Note You need to log in before you can comment on or make changes to this bug.