Bug 1490719 - Enabled ops cluser,log in kibana-ops UI, there is no log entry under .all index, log entries only could be shown under .operations.* index
Summary: Enabled ops cluser,log in kibana-ops UI, there is no log entry under .all ind...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.6.1
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.6.z
Assignee: Jeff Cantrill
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On: 1505683 1506099
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-09-12 06:31 UTC by Junqi Zhao
Modified: 2017-12-07 07:11 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2017-12-07 07:11:23 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
no log entries under .all index (109.03 KB, image/png)
2017-09-12 06:31 UTC, Junqi Zhao
no flags Details
log entries only could be found under .operatinns.* index (238.98 KB, image/png)
2017-09-12 06:32 UTC, Junqi Zhao
no flags Details
kibana UI, display .operations.* result (116.71 KB, image/png)
2017-10-17 11:29 UTC, Junqi Zhao
no flags Details
logging environment dump (62.31 KB, application/x-gzip)
2017-10-17 11:36 UTC, Junqi Zhao
no flags Details
kibana UI, it is noraml with logging-elasticsearch:v3.6.173.0.45-2 (275.12 KB, image/png)
2017-10-18 11:44 UTC, Junqi Zhao
no flags Details
kibana ops UI, no log entries under .all index (117.86 KB, image/png)
2017-10-18 11:45 UTC, Junqi Zhao
no flags Details
logging dump output, logging-elasticsearch is v3.6.173.0.45-2 (66.49 KB, application/x-gzip)
2017-10-18 11:47 UTC, Junqi Zhao
no flags Details
kibana-ops UI, there are logs under .all index (231.41 KB, image/png)
2017-11-22 09:02 UTC, Junqi Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github fabric8io openshift-elasticsearch-plugin pull 102 0 None None None 2017-09-21 22:22:09 UTC
Github openshift origin-aggregated-logging pull 675 0 None None None 2017-09-22 14:27:43 UTC
Red Hat Product Errata RHSA-2017:3389 0 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Enterprise security, bug fix, and enhancement update 2017-12-07 12:09:10 UTC

Description Junqi Zhao 2017-09-12 06:31:20 UTC
Created attachment 1324714 [details]
no log entries under .all index

Description of problem:
Deployed logging 3.6 with enabled ops cluser, log in kibana-ops UI,there is no log entry under .all index, log entries only could be shown under .operations.* index

Version-Release number of selected component (if applicable):
Images
logging-fluentd:v3.6.173.0.32-2
logging-kibana:v3.6.173.0.32-1
logging-elasticsearch:v3.6.173.0.32-1
logging-auth-proxy:v3.6.173.0.32-1
logging-curator:v3.6.173.0.32-1

How reproducible:
Always

Steps to Reproduce:
1. Deployed logging 3.6 with enabled ops cluser
2. Login kibana ops UI
3.

Actual results:
there is no log entry under .all index, log entries only could be shown under .operations.* index

Expected results:
there should be log entry under .all index

Additional info:
[OSEv3:children]
masters

[masters]
${MASTER_URL} openshift_public_hostname=${MASTER_URL}

[OSEv3:vars]
ansible_ssh_user=root
ansible_ssh_private_key_file="~/libra.pem"
deployment_type=openshift-enterprise

# Logging
openshift_logging_install_logging=true
openshift_logging_kibana_hostname=kibana.${SUB_DOMAIN}
openshift_logging_kibana_ops_hostname=kibana-ops.${SUB_DOMAIN}
public_master_url=https://${MASTER_URL}:${PORT}
openshift_logging_image_prefix=${IMAGE_PREFIX}
openshift_logging_image_version=v3.6
openshift_logging_namespace=logging
openshift_logging_use_ops=true
openshift_logging_elasticsearch_cpu_limit=500m
openshift_logging_elasticsearch_memory_limit=1Gi
openshift_logging_es_ops_cpu_limit=300m
openshift_logging_es_ops_memory_limit=512Mi

Comment 1 Junqi Zhao 2017-09-12 06:32:17 UTC
Created attachment 1324715 [details]
log entries only could be found under .operatinns.* index

Comment 2 Junqi Zhao 2017-09-12 06:33:36 UTC
Time range selected "Today"

Comment 5 Junqi Zhao 2017-10-12 02:31:24 UTC
Same error with https://bugzilla.redhat.com/show_bug.cgi?id=1499762

Comment 7 Junqi Zhao 2017-10-17 11:28:38 UTC
ops enabled cluster, there are no .all index in kibana-ops and kibana UI, for kibana-ops, it is ok, there is no error in the UI, but for kibana UI, when logging in kibana UI, the default kibana pages are all diplayed .operations.* results, this is wrong, and there is exception in UI: Discover: [exception] The index returned an empty result. You can use the Time Picker to change the time filter or select a higher time interval. See the attached pictures.

I think we should add .all index back for ops user.

Comment 8 Junqi Zhao 2017-10-17 11:29:45 UTC
Created attachment 1339668 [details]
kibana UI, display .operations.* result

Comment 9 Junqi Zhao 2017-10-17 11:36:57 UTC
Created attachment 1339669 [details]
logging environment dump

Comment 15 Junqi Zhao 2017-10-18 11:44:17 UTC
Created attachment 1340134 [details]
kibana UI, it is noraml with logging-elasticsearch:v3.6.173.0.45-2

Comment 16 Junqi Zhao 2017-10-18 11:45:52 UTC
Created attachment 1340135 [details]
kibana ops UI, no log entries under .all index

Comment 17 Junqi Zhao 2017-10-18 11:47:54 UTC
Created attachment 1340136 [details]
logging dump output, logging-elasticsearch is  v3.6.173.0.45-2

Comment 20 Junqi Zhao 2017-10-25 02:43:47 UTC
Blocks by https://bugzilla.redhat.com/show_bug.cgi?id=1505683

Comment 21 Anping Li 2017-10-30 06:21:02 UTC
Test with logging: v3.6.173.0.62. operations.* index appears in regurlar kibana, and Exception [1] appears.  

Only the .operations.* index is in kibana-ops.  I remember that  there should be  an all item. When and why we drop the all item?

[1] Exception in regurlar kibana.
Error: [exception] The index returned an empty result. You can use the Time Picker to change the time filter or select a higher time interval
ErrorAbstract@https://kibana.apps.1030-knj.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:95501:20
StatusCodeError@https://kibana.apps.1030-knj.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:95663:6
respond@https://kibana.apps.1030-knj.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:96868:16
checkRespForFailure@https://kibana.apps.1030-knj.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:96831:8
[24]</AngularConnector.prototype.request/<@https://kibana.apps.1030-knj.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:95449:8
processQueue@https://kibana.apps.1030-knj.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:42452:29
scheduleProcessQueue/<@https://kibana.apps.1030-knj.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:42468:28
$RootScopeProvider/this.$get</Scope.prototype.$eval@https://kibana.apps.1030-knj.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43696:17
$RootScopeProvider/this.$get</Scope.prototype.$digest@https://kibana.apps.1030-knj.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43507:16
$RootScopeProvider/this.$get</Scope.prototype.$apply@https://kibana.apps.1030-knj.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43804:14
done@https://kibana.apps.1030-knj.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38253:37
completeRequest@https://kibana.apps.1030-knj.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38451:8
requestLoaded@https://kibana.apps.1030-knj.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38392:10

Comment 22 Anping Li 2017-11-16 10:48:37 UTC
There is .all index in v3.6.173.0.63. but I got message "No results found"  when click .all index.

Comment 23 Jeff Cantrill 2017-11-16 13:33:59 UTC
@Anping,

* Can you please query ES directly to confirm you have data? Dumping the indices will show you a document count
* Do you have a time range that includes data?

Comment 25 Junqi Zhao 2017-11-22 09:01:55 UTC
Tested with v3.6.173.0.78-1 logging images, deployed logging with enabled ops cluster, There are logs under .all index in kibana-ops UI. See the snapshot

Please set this defect as ON_QA, so we can close it.

Comment 26 Junqi Zhao 2017-11-22 09:02:31 UTC
Created attachment 1357314 [details]
kibana-ops UI, there are logs under .all index

Comment 27 Junqi Zhao 2017-11-23 00:31:32 UTC
Set this defect to VERIFIED as per Comment 25

Comment 30 errata-xmlrpc 2017-12-07 07:11:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3389


Note You need to log in before you can comment on or make changes to this bug.