Bug 1571190 - write kube-* namespace logs to .operations index in v3.10
Summary: write kube-* namespace logs to .operations index in v3.10
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.11.0
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
: 1571478 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-24 09:22 UTC by Anping Li
Modified: 2019-06-26 09:08 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: Archive namespace logs for kube* to the operations index Reason: These are operations namespaces Result: Logs are indexed under .operations
Clone Of:
Environment:
Last Closed: 2019-06-26 09:07:51 UTC
Target Upstream Version:


Attachments (Terms of Use)
No kube-system in index message count widget. (18.40 KB, image/png)
2018-05-03 19:32 UTC, Mike Fiedler
no flags Details
kube-system search returns no records (43.38 KB, image/png)
2018-05-03 19:34 UTC, Mike Fiedler
no flags Details
kube-system search returns no records (75.49 KB, image/png)
2018-05-03 20:06 UTC, Mike Fiedler
no flags Details


Links
System ID Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 1119 None None None 2018-04-25 18:05:44 UTC
Red Hat Product Errata RHBA-2019:1605 None None None 2019-06-26 09:07:59 UTC

Description Anping Li 2018-04-24 09:22:55 UTC
Description of problem:
In v3.10, some infra components are deployed in the dedicated projects.  The ENV OCP_OPERATIONS_PROJECTS control the .operations index projects.  The default value is "default openshift openshift-".  I think kube-system should be added as default.  As logging namespaces changed from logging to openshift-logging, it will be added the .operations index. Is that expected?

The etcd and masters are deployed in kube-system project. 
The Logging are deployed in openshift-logging project. 
The sync pod are deployed in openshift-node projects.
The ovs and sdn pods are deployed in openshift-sdn. 

Version-Release number of selected component (if applicable):
OCP 3.10

How reproducible:
always

Steps to Reproduce:
1. Deploy OCP 3.10
2. check the infra component projects

# oc get projects
NAME                    DISPLAY NAME   STATUS
default                                Active
kube-public                            Active
kube-system                            Active
management-infra                       Active
openshift                              Active
openshift-infra                        Active
openshift-logging                      Active
openshift-metrics                      Active
openshift-node                         Active
openshift-web-console                  Active

3. check the default values of OCP_OPERATIONS_PROJECTS in fluentd

Actual results:

OCP_OPERATIONS_PROJECTS=${OCP_OPERATIONS_PROJECTS:-"default openshift openshift-"}


Expected results:


OCP_OPERATIONS_PROJECTS=${OCP_OPERATIONS_PROJECTS:-"default openshift kube- openshift-"}


Additional info:

Comment 1 Rich Megginson 2018-04-24 15:09:40 UTC
I think what you are asking is that logs from kube-* namespaces should go into the .operations index - is that correct?

Comment 2 Anping Li 2018-04-25 02:05:10 UTC
@rich, Yes.

Comment 3 Mike Fiedler 2018-04-26 22:34:28 UTC
*** Bug 1571478 has been marked as a duplicate of this bug. ***

Comment 4 Mike Fiedler 2018-05-03 14:46:05 UTC
In my opinion, losing access to master controller, master api and etcd logs in kibana is a regression.   In 3.9, customers using our recommendation to use the json-file logdriver would get master/etcd logs in kibana via .operations logs.

In 3.10, if the json-file driver is used, the customers lose the ability to search/view master and etcd logs in kibana.  I believe this to be a bug that needs to be addressed in 3.10 which is why I opened https://bugzilla.redhat.com/show_bug.cgi?id=1571478 as a bug, not RFE.  It was "dup-ed" against this bz.

Comment 5 Rich Megginson 2018-05-03 18:24:08 UTC
(In reply to Mike Fiedler from comment #4)
> In my opinion, losing access to master controller, master api and etcd logs
> in kibana is a regression.   In 3.9, customers using our recommendation to
> use the json-file logdriver would get master/etcd logs in kibana via
> .operations logs.
> 
> In 3.10, if the json-file driver is used, the customers lose the ability to
> search/view master and etcd logs in kibana.  I believe this to be a bug that
> needs to be addressed in 3.10 which is why I opened
> https://bugzilla.redhat.com/show_bug.cgi?id=1571478 as a bug, not RFE.  It
> was "dup-ed" against this bz.

I don't understand - we're not losing anything?

Comment 6 Mike Fiedler 2018-05-03 19:31:58 UTC
Re:  comment 5.  We are not losing anything - it is in ES.  It is not accessible (at least OOTB) in the kibana UI.  It does not appear in the drop down list of indices and it does not appear under the kubernetes.namespace_name graph in the left hand navigation (screenshot #1).

None of the records appear when I enter kubernetes.namespace_name:kube_system in the search field (screenshot #2).

1.  Indices:

health status index                                                                        pri rep docs.count docs.deleted store.size pri.store.size                                                                                                                                      
green  open   .operations.2018.05.03                                                         1   0      73823            0     65.7mb         65.7mb                                                                                                                                      
green  open   .searchguard.logging-es-data-master-cjnsnppx                                   1   2          5            0    103.3kb         34.4kb                                                                                                                                      
green  open   .kibana                                                                        1   0          1            0      3.1kb          3.1kb                                                                                                                                      
green  open   project.kube-system.5d3a8835-4ef9-11e8-b03b-0206b107b3f6.2018.05.03            1   0      28147            0     11.3mb         11.3mb                                                                                                                                      
green  open   .searchguard.logging-es-data-master-0nt5pfad                                   1   2          5            0    103.3kb         34.4kb                                                                                                                                      
green  open   project.kube-service-catalog.0f865cd9-4efa-11e8-b03b-0206b107b3f6.2018.05.03   1   0       5267            0      1.9mb          1.9mb                                                                                                                                      
green  open   .searchguard.logging-es-data-master-k1vd11db                                   1   2          5            0    103.3kb         34.4kb  


2.  Hitting the ES REST API to search this index returns good data.  When I hit https://logging-es:9200/_search?q=kubernetes.namespace_name:kube-system  I get data back.

3. Searching kubernetes.namespace_name:kube-system returns no data.  See screenshots 1 and 2.   The kube-system entry also does not appear in the index dropdown (see bz 1571478)

Comment 7 Mike Fiedler 2018-05-03 19:32:37 UTC
Created attachment 1430869 [details]
No kube-system in index message count widget.

Comment 8 Mike Fiedler 2018-05-03 19:34:09 UTC
Created attachment 1430871 [details]
kube-system search returns no records

Comment 9 Rich Megginson 2018-05-03 19:43:51 UTC
(In reply to Mike Fiedler from comment #8)
> Created attachment 1430871 [details]
> kube-system search returns no records

try kube-system instead of kube_system

Comment 10 Mike Fiedler 2018-05-03 20:06:40 UTC
Created attachment 1430878 [details]
kube-system search returns no records

sorry for the typo.  kube-system returns no records.  I verified that searching something like kubernetes.namespace_name:openshift-logging does return records.

Comment 11 openshift-github-bot 2018-07-06 19:24:33 UTC
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/dc3d434fb5fac29ba9b448b4985264c92c7b02d3
bug 1571190. Add kube-system as an operations project

Comment 15 Rich Megginson 2018-10-02 20:57:46 UTC
If you go to the Discover tab in Kibana, on the left hand side (not in the index drop down) there is a list of fields.  One of the fields is namespace_name or kubernetes.namespace_name.  If you click on this, is "kube-system" listed as one of the values?  If not, if you expand the time window using the time picker, does it show up?

In the query field, can you type kubernetes.namespace_name:kube-system and see the logs?

Comment 16 Tushar Katarki 2019-05-17 20:29:06 UTC
Why is this being tracked as an RFE? It appears to be a regression. Please re-file as a bug.

Comment 17 Mike Fiedler 2019-05-20 12:34:28 UTC
Re-opening as a bug per comment 16.

Comment 19 Anping Li 2019-06-12 16:00:10 UTC
verified in logging-fluentd:v3.11.117

Comment 21 errata-xmlrpc 2019-06-26 09:07:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1605


Note You need to log in before you can comment on or make changes to this bug.