Bug 1652224 - After upgrade 3.10 to 3.11 cluster-admin can't browse Elasticsearch logs from kibana
Summary: After upgrade 3.10 to 3.11 cluster-admin can't browse Elasticsearch logs from...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.11.z
Assignee: Jeff Cantrill
QA Contact: Anping Li
Depends On:
TreeView+ depends on / blocked
Reported: 2018-11-21 17:26 UTC by mmariyan
Modified: 2022-03-13 16:10 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Improper evaluation of a user's Kibana index Consequence: Minor upgrade in server version causes an error when the expected configuration object was not because its creation was skipped due to the existence of kibana index Fix: Work around is to remove a user's Kiana index. The proper fix is to evaluate the stored version against the Kibana version and recreate the config object if necessary. Result: User's will no longer see the error
Clone Of:
Last Closed: 2019-01-10 09:04:12 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 1478 0 'None' closed bug 1652224. Fix defaultIndex for Kibana uprade 2021-02-12 09:10:33 UTC
Red Hat Knowledge Base (Solution) 3722751 0 None None None 2018-12-01 00:20:32 UTC
Red Hat Product Errata RHBA-2019:0024 0 None None None 2019-01-10 09:05:49 UTC

Description mmariyan 2018-11-21 17:26:57 UTC
Description of problem:

 depolyed OSCP 3.10 and everything was working fine but after upgrading to 3.11 user with cluster-admin role is unable to view logs in kibana, just like in:


Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. deploy 3.10 cluster
2. user with cluster-admin role login to kibana (everything is fine)
3. upgrade cluster
4. the same user can't view logs

Expected results:

Additional info:

Comment 4 Jeff Cantrill 2018-11-26 14:05:19 UTC
Continuing to investigate but the short term fix would be to remove the kibana index for the particular user.  I would recommend:

1. Export any custom index-patterns, visualizations, dashboards
2. Find the user's index pattern [1]
3. Delete the user's index pattern:

oc exec -c elasticsearch $pod -- es_util --query=$KIBANA_INDEX -XDELETE

[1] https://github.com/jcantrill/cluster-logging-tools/blob/master/scripts/kibana-index-name

Comment 5 Jeff Cantrill 2018-11-29 19:03:10 UTC
Issue is because we skip [1] any additional logic if index already exists.  Need to take into account version.  Work around is #c4

[1] https://github.com/fabric8io/openshift-elasticsearch-plugin/blob/master/src/main/java/io/fabric8/elasticsearch/plugin/kibana/KibanaSeed.java#L63

Comment 6 Robert Bost 2018-11-29 23:53:32 UTC
Adding some clarification to Jeff's workaround:

> 2. Find the user's index pattern [1]
> [1] https://github.com/jcantrill/cluster-logging-tools/blob/master/scripts/kibana-index-name

Make sure to add leading period to ".kibana.xxxxxx" when deleting the index.


Comment 7 Jeff Cantrill 2018-12-05 19:11:46 UTC
Fixed in https://github.com/openshift/origin-aggregated-logging/pull/1478

Comment 8 openshift-github-bot 2018-12-14 17:11:32 UTC
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging

bug 1652224. Fix defaultIndex for Kibana uprade

Merge pull request #1481 from openshift-cherrypick-robot/cherry-pick-1478-to-master

[master] bug 1652224. Fix defaultIndex for Kibana uprade

Comment 13 Anping Li 2018-12-21 08:59:42 UTC
The user continue to login Kibana once upgrade from v3.10 to v3.11.59

Comment 15 errata-xmlrpc 2019-01-10 09:04:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.