Bug 1652224

Summary: After upgrade 3.10 to 3.11 cluster-admin can't browse Elasticsearch logs from kibana
Product: OpenShift Container Platform Reporter: mmariyan
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: high Docs Contact:
Priority: high    
Version: 3.11.0CC: aos-bugs, cshereme, rbost, rmeggins, sauchter, stwalter
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Improper evaluation of a user's Kibana index Consequence: Minor upgrade in server version causes an error when the expected configuration object was not because its creation was skipped due to the existence of kibana index Fix: Work around is to remove a user's Kiana index. The proper fix is to evaluate the stored version against the Kibana version and recreate the config object if necessary. Result: User's will no longer see the error
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-10 09:04:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description mmariyan 2018-11-21 17:26:57 UTC
Description of problem:

 depolyed OSCP 3.10 and everything was working fine but after upgrading to 3.11 user with cluster-admin role is unable to view logs in kibana, just like in:

https://github.com/openshift/origin-aggregated-logging/issues/1333

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. deploy 3.10 cluster
2. user with cluster-admin role login to kibana (everything is fine)
3. upgrade cluster
4. the same user can't view logs

Expected results:


Additional info:

Comment 4 Jeff Cantrill 2018-11-26 14:05:19 UTC
Continuing to investigate but the short term fix would be to remove the kibana index for the particular user.  I would recommend:

1. Export any custom index-patterns, visualizations, dashboards
2. Find the user's index pattern [1]
3. Delete the user's index pattern:

oc exec -c elasticsearch $pod -- es_util --query=$KIBANA_INDEX -XDELETE

[1] https://github.com/jcantrill/cluster-logging-tools/blob/master/scripts/kibana-index-name

Comment 5 Jeff Cantrill 2018-11-29 19:03:10 UTC
Issue is because we skip [1] any additional logic if index already exists.  Need to take into account version.  Work around is #c4

[1] https://github.com/fabric8io/openshift-elasticsearch-plugin/blob/master/src/main/java/io/fabric8/elasticsearch/plugin/kibana/KibanaSeed.java#L63

Comment 6 Robert Bost 2018-11-29 23:53:32 UTC
Adding some clarification to Jeff's workaround:

> 2. Find the user's index pattern [1]
> [1] https://github.com/jcantrill/cluster-logging-tools/blob/master/scripts/kibana-index-name

Make sure to add leading period to ".kibana.xxxxxx" when deleting the index.

https://github.com/jcantrill/cluster-logging-tools/pull/3

Comment 7 Jeff Cantrill 2018-12-05 19:11:46 UTC
Fixed in https://github.com/openshift/origin-aggregated-logging/pull/1478

Comment 8 openshift-github-bot 2018-12-14 17:11:32 UTC
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/b275e32d7db20201e075d5435aeb7de650660c30
bug 1652224. Fix defaultIndex for Kibana uprade

https://github.com/openshift/origin-aggregated-logging/commit/6d5d0b7e8d9fe4826b602f14489c21ad5ca14ea5
Merge pull request #1481 from openshift-cherrypick-robot/cherry-pick-1478-to-master

[master] bug 1652224. Fix defaultIndex for Kibana uprade

Comment 13 Anping Li 2018-12-21 08:59:42 UTC
The user continue to login Kibana once upgrade from v3.10 to v3.11.59

Comment 15 errata-xmlrpc 2019-01-10 09:04:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0024