Description of problem: When fluentd is not able to find namespace or pod metadata it pushes that message to '.orphaned' index. Currently there is no way to see that index in kibana until es pod settings are modified manually. Version-Release number of selected component (if applicable): openshift v3.7.9 kubernetes v1.7.6+a08f5eeb62 etcd 3.2.8 How reproducible: when someone need to look for messages in kibana under orphaned index. Steps to Reproduce: 1. Create few pods which are logging at high rate (currently i was able to do it @ 1250 messages/mux/sec) 2. delete projects 3. see indexes, there will be a index project..orphaned.orphaned* Actual results: Messages are not visible in kibana. Expected results: Admins should be able to see logs in kibana Additional info:
Pushing to 3.10 as the filter code still is not published in the code base.
Created attachment 1394934 [details] kibana and orphan messages
This will be resolved by https://github.com/openshift/origin-aggregated-logging/pull/1048
The orphaned index can be view by cluster-admin in v3.9.27
.project.* and .orphaned index pattern are set by default
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1566