Bug 1571190
| Summary: | write kube-* namespace logs to .operations index in v3.10 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Anping Li <anli> | ||||||||
| Component: | Logging | Assignee: | Jeff Cantrill <jcantril> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | high | ||||||||||
| Version: | 3.10.0 | CC: | aos-bugs, jcantril, jmalde, jokerman, mifiedle, mmccomas, mrobson, rmeggins, tkatarki | ||||||||
| Target Milestone: | --- | Keywords: | Reopened | ||||||||
| Target Release: | 3.11.0 | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | Enhancement | |||||||||
| Doc Text: |
Feature: Archive namespace logs for kube* to the operations index
Reason: These are operations namespaces
Result: Logs are indexed under .operations
|
Story Points: | --- | ||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2019-06-26 09:07:51 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
I think what you are asking is that logs from kube-* namespaces should go into the .operations index - is that correct? @rich, Yes. *** Bug 1571478 has been marked as a duplicate of this bug. *** In my opinion, losing access to master controller, master api and etcd logs in kibana is a regression. In 3.9, customers using our recommendation to use the json-file logdriver would get master/etcd logs in kibana via .operations logs. In 3.10, if the json-file driver is used, the customers lose the ability to search/view master and etcd logs in kibana. I believe this to be a bug that needs to be addressed in 3.10 which is why I opened https://bugzilla.redhat.com/show_bug.cgi?id=1571478 as a bug, not RFE. It was "dup-ed" against this bz. (In reply to Mike Fiedler from comment #4) > In my opinion, losing access to master controller, master api and etcd logs > in kibana is a regression. In 3.9, customers using our recommendation to > use the json-file logdriver would get master/etcd logs in kibana via > .operations logs. > > In 3.10, if the json-file driver is used, the customers lose the ability to > search/view master and etcd logs in kibana. I believe this to be a bug that > needs to be addressed in 3.10 which is why I opened > https://bugzilla.redhat.com/show_bug.cgi?id=1571478 as a bug, not RFE. It > was "dup-ed" against this bz. I don't understand - we're not losing anything? Re: comment 5. We are not losing anything - it is in ES. It is not accessible (at least OOTB) in the kibana UI. It does not appear in the drop down list of indices and it does not appear under the kubernetes.namespace_name graph in the left hand navigation (screenshot #1). None of the records appear when I enter kubernetes.namespace_name:kube_system in the search field (screenshot #2). 1. Indices: health status index pri rep docs.count docs.deleted store.size pri.store.size green open .operations.2018.05.03 1 0 73823 0 65.7mb 65.7mb green open .searchguard.logging-es-data-master-cjnsnppx 1 2 5 0 103.3kb 34.4kb green open .kibana 1 0 1 0 3.1kb 3.1kb green open project.kube-system.5d3a8835-4ef9-11e8-b03b-0206b107b3f6.2018.05.03 1 0 28147 0 11.3mb 11.3mb green open .searchguard.logging-es-data-master-0nt5pfad 1 2 5 0 103.3kb 34.4kb green open project.kube-service-catalog.0f865cd9-4efa-11e8-b03b-0206b107b3f6.2018.05.03 1 0 5267 0 1.9mb 1.9mb green open .searchguard.logging-es-data-master-k1vd11db 1 2 5 0 103.3kb 34.4kb 2. Hitting the ES REST API to search this index returns good data. When I hit https://logging-es:9200/_search?q=kubernetes.namespace_name:kube-system I get data back. 3. Searching kubernetes.namespace_name:kube-system returns no data. See screenshots 1 and 2. The kube-system entry also does not appear in the index dropdown (see bz 1571478) Created attachment 1430869 [details]
No kube-system in index message count widget.
Created attachment 1430871 [details]
kube-system search returns no records
(In reply to Mike Fiedler from comment #8) > Created attachment 1430871 [details] > kube-system search returns no records try kube-system instead of kube_system Created attachment 1430878 [details]
kube-system search returns no records
sorry for the typo. kube-system returns no records. I verified that searching something like kubernetes.namespace_name:openshift-logging does return records.
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/dc3d434fb5fac29ba9b448b4985264c92c7b02d3 bug 1571190. Add kube-system as an operations project If you go to the Discover tab in Kibana, on the left hand side (not in the index drop down) there is a list of fields. One of the fields is namespace_name or kubernetes.namespace_name. If you click on this, is "kube-system" listed as one of the values? If not, if you expand the time window using the time picker, does it show up? In the query field, can you type kubernetes.namespace_name:kube-system and see the logs? Why is this being tracked as an RFE? It appears to be a regression. Please re-file as a bug. Re-opening as a bug per comment 16. verified in logging-fluentd:v3.11.117 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1605 |
Description of problem: In v3.10, some infra components are deployed in the dedicated projects. The ENV OCP_OPERATIONS_PROJECTS control the .operations index projects. The default value is "default openshift openshift-". I think kube-system should be added as default. As logging namespaces changed from logging to openshift-logging, it will be added the .operations index. Is that expected? The etcd and masters are deployed in kube-system project. The Logging are deployed in openshift-logging project. The sync pod are deployed in openshift-node projects. The ovs and sdn pods are deployed in openshift-sdn. Version-Release number of selected component (if applicable): OCP 3.10 How reproducible: always Steps to Reproduce: 1. Deploy OCP 3.10 2. check the infra component projects # oc get projects NAME DISPLAY NAME STATUS default Active kube-public Active kube-system Active management-infra Active openshift Active openshift-infra Active openshift-logging Active openshift-metrics Active openshift-node Active openshift-web-console Active 3. check the default values of OCP_OPERATIONS_PROJECTS in fluentd Actual results: OCP_OPERATIONS_PROJECTS=${OCP_OPERATIONS_PROJECTS:-"default openshift openshift-"} Expected results: OCP_OPERATIONS_PROJECTS=${OCP_OPERATIONS_PROJECTS:-"default openshift kube- openshift-"} Additional info: