Bug 1920828

Description of problem:

Hi,

I've configured Elasticsearch log Curator to delete 1 day older indexed  data of user defined project(eg: myproj-qe) and observed curator could not  delete  "myproj-qe" project data which is older than 1 day.
 
Could you please check is below steps are correct as per doc.? and should it create indices as format  "projectName-timestamp" in ES and later can be rollover by Curator by age?

reference  RH links:-

https://access.redhat.com/documentation/en-us/openshift_container_platform/4.5/html/release_notes/ocp-4-5-release-notes
https://docs.openshift.com/container-platform/4.6/logging/config/cluster-logging-curator.html

Version-Release number of selected component (if applicable):
#oc version
Client Version: 4.7.0-fc.2
Server Version: 4.7.0-fc.2
Kubernetes Version: v1.20.0+394a5a3

How reproducible:


Steps to Reproduce:
1. Setup cluster logging instance 
2. create new project("myproj-qe") and run some sample network workload(iperf3) to produce logs as application log for ES app index.

# oc get pods -n myproj-qe
NAME                          READY   STATUS    RESTARTS   AGE
network-stress-client-5v9gn   1/1     Running   0          19h
network-stress-server-q4mmd   1/1     Running   0          19h


3. edit Curator configuration file- oc edit configmap/curator. and add below in config.yaml: section
   myproj-qe:
      delete:
        days: 1
4.checked the Kibana application log after 1 day, it still showing data of  "myproj-qe" project in "app" index, which is older than 1 day.

Actual results:


Expected results:


Additional info:
NOTES:  we have enabled the debug log of curator, and observed, the  "filtertype: age" is calculated based on "Source"(eg. indices name) and  "timestring" associated with the indices name(eg. curator expecting indices format-  myproj-%Y.%m.%d),  But in OCP the index name generated by ES are generic in nature (e.g., infra-000001 or app-000003 or audit-000021) and Logs from all application namespaces are stored in a given app-###### index name. Also we could not see any index format( eg. myproj-qe-### ) in ES.

OCP ES index format :
--------------------
# oc exec elasticsearch-cdm-ofr1aa67-1-bf99cb4dd-r56p9  -- indices
health status index                          uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   app-000003                     xekH_V4NSGSjHV0n91Tx6Q   1   0      14108            0          5              5
green  open   audit-000021                   uXocn7PITF6kukm3TKvxfQ   1   0     567615            0        453            453
green  open   infra-000021                   AFtdNCaYRjyW_777AfG7eQ   1   0     190334            0        119            119

some lines of Curator DEBUG logs(log details attached )
===============================
...
+ cat /opt/app-root/src/actions.yaml
---
actions:
  1:
    action: delete_indices
    description: auto-generated
    options:
      ignore_empty_list: true
      continue_if_exception: false
      timeout_override: '300'
    filters:
    - filtertype: pattern
      kind: regex
      value: '^project\.myproj\-qe\..*$'
      exclude: false
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 1
......

2021-01-22 09:00:05,535 INFO                 curator.cli                    run:148  Preparing Action ID: 1, "delete_indices"
...
2021-01-22 09:00:06,055 DEBUG                curator.cli         process_action:93   Running "DELETE_INDICES"
2021-01-22 09:00:06,055 DEBUG          curator.indexlist          __get_indices:65   Getting all indices
2021-01-22 09:00:06,075 DEBUG              curator.utils            get_indices:648  Detected Elasticsearch version 6.8.1
2021-01-22 09:00:06,076 DEBUG              curator.utils            get_indices:650  All indices: ['infra-000190', 'infra-000197', 'app-000039', 'app-000030', 'infra-000205', 'infra-000193', 'infra-000192', 'audit-000261', 'app-000038', 'infra-000195', '.kibana_-377444158_kubeadmin', 'app-000024', 'app-000020', 'infra-000188', 'app-000027', 'audit-000251', 'infra-000203', 'audit-000260', 'app-000023', 'infra-000206', '.security', 'app-000040', 'app-000018', 'infra-000200', 'audit-000256', 'infra-000191', 'app-000036', 'infra-000204', 'app-000017', 'infra-000198', 'audit-000258', 'infra-000187', 'audit-000257', 'infra-000189', 'app-000019', 'app-000028', 'app-000034', 'infra-000196', 'app-000037', 'audit-000254', 'audit-000250', 'app-000029', 'app-000025', 'infra-000194', '.kibana_1', 'audit-000262', 'audit-000263', 'app-000032', 'app-000031', 'app-000021', 'audit-000259', 'audit-000255', 'infra-000201', 'infra-000202', 'audit-000253', 'app-000026', 'app-000022', 'app-000035', 'audit-000252', 'app-000033', 'infra-000199']
...
2021-01-22 09:00:05,972 DEBUG          curator.indexlist          filter_by_age:490  Filtering indices by age
2021-01-22 09:00:05,972 DEBUG          curator.indexlist   _get_name_based_ages:280  Getting ages of indices by "name"
2021-01-22 09:00:05,972 DEBUG          curator.indexlist       empty_list_check:226  Checking for empty list
2021-01-22 09:00:05,972 INFO                 curator.cli                    run:202  Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2021-01-22 09:00:05,972 INFO                 curator.cli                    run:223  Action ID: 1, "delete_indices" completed.
2021-01-22 09:00:05,972 DEBUG                curator.cli                    run:128  action_disabled = False
2021-01-22 09:00:05,972 DEBUG                curator.cli                    run:132  continue_if_exception = False
2021-01-22 09:00:05,972 DEBUG                curator.cli                    run:134  timeout_override = 300
2021-01-22 09:00:05,972 DEBUG                curator.cli                    run:136  ignore_empty_list = True
2021-01-22 09:00:05,972 DEBUG                curator.cli                    run:138  allow_ilm_indices = False
2021-01-22 09:00:05,973 INFO                 curator.cli                    run:148  Preparing Action ID: 2, "delete_indices"