Description of problem: In v3.10, some infra components are deployed in the dedicated projects. The ENV OCP_OPERATIONS_PROJECTS control the .operations index projects. The default value is "default openshift openshift-". I think kube-system should be added as default. As logging namespaces changed from logging to openshift-logging, it will be added the .operations index. Is that expected? The etcd and masters are deployed in kube-system project. The Logging are deployed in openshift-logging project. The sync pod are deployed in openshift-node projects. The ovs and sdn pods are deployed in openshift-sdn. Version-Release number of selected component (if applicable): OCP 3.10 How reproducible: always Steps to Reproduce: 1. Deploy OCP 3.10 2. check the infra component projects # oc get projects NAME DISPLAY NAME STATUS default Active kube-public Active kube-system Active management-infra Active openshift Active openshift-infra Active openshift-logging Active openshift-metrics Active openshift-node Active openshift-web-console Active 3. check the default values of OCP_OPERATIONS_PROJECTS in fluentd Actual results: OCP_OPERATIONS_PROJECTS=${OCP_OPERATIONS_PROJECTS:-"default openshift openshift-"} Expected results: OCP_OPERATIONS_PROJECTS=${OCP_OPERATIONS_PROJECTS:-"default openshift kube- openshift-"} Additional info:
I think what you are asking is that logs from kube-* namespaces should go into the .operations index - is that correct?
@rich, Yes.
*** Bug 1571478 has been marked as a duplicate of this bug. ***
In my opinion, losing access to master controller, master api and etcd logs in kibana is a regression. In 3.9, customers using our recommendation to use the json-file logdriver would get master/etcd logs in kibana via .operations logs. In 3.10, if the json-file driver is used, the customers lose the ability to search/view master and etcd logs in kibana. I believe this to be a bug that needs to be addressed in 3.10 which is why I opened https://bugzilla.redhat.com/show_bug.cgi?id=1571478 as a bug, not RFE. It was "dup-ed" against this bz.
(In reply to Mike Fiedler from comment #4) > In my opinion, losing access to master controller, master api and etcd logs > in kibana is a regression. In 3.9, customers using our recommendation to > use the json-file logdriver would get master/etcd logs in kibana via > .operations logs. > > In 3.10, if the json-file driver is used, the customers lose the ability to > search/view master and etcd logs in kibana. I believe this to be a bug that > needs to be addressed in 3.10 which is why I opened > https://bugzilla.redhat.com/show_bug.cgi?id=1571478 as a bug, not RFE. It > was "dup-ed" against this bz. I don't understand - we're not losing anything?
Re: comment 5. We are not losing anything - it is in ES. It is not accessible (at least OOTB) in the kibana UI. It does not appear in the drop down list of indices and it does not appear under the kubernetes.namespace_name graph in the left hand navigation (screenshot #1). None of the records appear when I enter kubernetes.namespace_name:kube_system in the search field (screenshot #2). 1. Indices: health status index pri rep docs.count docs.deleted store.size pri.store.size green open .operations.2018.05.03 1 0 73823 0 65.7mb 65.7mb green open .searchguard.logging-es-data-master-cjnsnppx 1 2 5 0 103.3kb 34.4kb green open .kibana 1 0 1 0 3.1kb 3.1kb green open project.kube-system.5d3a8835-4ef9-11e8-b03b-0206b107b3f6.2018.05.03 1 0 28147 0 11.3mb 11.3mb green open .searchguard.logging-es-data-master-0nt5pfad 1 2 5 0 103.3kb 34.4kb green open project.kube-service-catalog.0f865cd9-4efa-11e8-b03b-0206b107b3f6.2018.05.03 1 0 5267 0 1.9mb 1.9mb green open .searchguard.logging-es-data-master-k1vd11db 1 2 5 0 103.3kb 34.4kb 2. Hitting the ES REST API to search this index returns good data. When I hit https://logging-es:9200/_search?q=kubernetes.namespace_name:kube-system I get data back. 3. Searching kubernetes.namespace_name:kube-system returns no data. See screenshots 1 and 2. The kube-system entry also does not appear in the index dropdown (see bz 1571478)
Created attachment 1430869 [details] No kube-system in index message count widget.
Created attachment 1430871 [details] kube-system search returns no records
(In reply to Mike Fiedler from comment #8) > Created attachment 1430871 [details] > kube-system search returns no records try kube-system instead of kube_system
Created attachment 1430878 [details] kube-system search returns no records sorry for the typo. kube-system returns no records. I verified that searching something like kubernetes.namespace_name:openshift-logging does return records.
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/dc3d434fb5fac29ba9b448b4985264c92c7b02d3 bug 1571190. Add kube-system as an operations project
If you go to the Discover tab in Kibana, on the left hand side (not in the index drop down) there is a list of fields. One of the fields is namespace_name or kubernetes.namespace_name. If you click on this, is "kube-system" listed as one of the values? If not, if you expand the time window using the time picker, does it show up? In the query field, can you type kubernetes.namespace_name:kube-system and see the logs?
Why is this being tracked as an RFE? It appears to be a regression. Please re-file as a bug.
Re-opening as a bug per comment 16.
verified in logging-fluentd:v3.11.117
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1605