Bug 1656086 - Users unable to access Kibana after upgrade to 3.11
Summary: Users unable to access Kibana after upgrade to 3.11
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 3.11.z
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-04 16:26 UTC by Matthew Barnes
Modified: 2019-04-10 06:09 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Bug in earlier releases of cluster-logging introduced Kibana index-patterns where the title was not properly replaced and was left with the place holder of '$TITLE$' Consequence: User sees permission error 'no permissions for [indices:data/read/field_caps]' Fix: Remove all index-patterns which have the bad data either by upgrading or running 'oc exec -c elasticsearch -n $NS $pod -- es_util --query=".kibana.*/_delete_by_query?pretty" -d "{\"query\":{\"match\":{\"title\":\"*TITLE*\"}}}"' Result: The permission error is no longer generated
Clone Of:
Environment:
Last Closed: 2019-01-10 09:04:12 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 1482 'None' closed bug 1656086. remove index-patterns with title=$TITLE 2020-02-11 02:08:46 UTC
Github openshift origin-aggregated-logging pull 1485 'None' closed bug 1656086. code review fixes 2020-02-11 02:08:46 UTC
Red Hat Product Errata RHBA-2019:0024 None None None 2019-01-10 09:05:49 UTC

Description Matthew Barnes 2018-12-04 16:26:58 UTC
After upgrading to OpenShift 3.11, users are encountering permission errors when trying to access Kibana.

Error: [object Object]: [security_exception] no permissions for [indices:data/read/field_caps] and User [name=USERNAME, roles=[gen_user_efad75b1cdea4c3c1e42f8bf5fbeefdde90c0fb0, gen_kibana_efad75b1cdea4c3c1e42f8bf5fbeefdde90c0fb0]]
    at https://logs.insights.openshift.com/bundles/kibana.bundle.js?v=15653:227:21353
    at processQueue (https://logs.insights.openshift.com/bundles/commons.bundle.js?v=15653:38:23621)
    at https://logs.insights.openshift.com/bundles/commons.bundle.js?v=15653:38:23888
    at Scope.$eval (https://logs.insights.openshift.com/bundles/commons.bundle.js?v=15653:39:4619)
    at Scope.$digest (https://logs.insights.openshift.com/bundles/commons.bundle.js?v=15653:39:2359)
    at Scope.$apply (https://logs.insights.openshift.com/bundles/commons.bundle.js?v=15653:39:5037)
    at done (https://logs.insights.openshift.com/bundles/commons.bundle.js?v=15653:37:25027)
    at completeRequest (https://logs.insights.openshift.com/bundles/commons.bundle.js?v=15653:37:28702)
    at XMLHttpRequest.xhr.onload (https://logs.insights.openshift.com/bundles/commons.bundle.js?v=15653:37:29634)

Short term workaround from bug https://bugzilla.redhat.com/show_bug.cgi?id=1652224#c4 applies here as well.

Comment 1 Jeff Cantrill 2018-12-05 19:18:22 UTC
I believe this is a result of bad index-pattern seeding left over from 3.9 that is only manifesting itself now.  It appears there are issues with any index-pattern that has source with 'title' equal to '$TITLE'.  This was the placeholder for our index-pattern template.

Work around is to:
a. Delete the kibana index pattern

or
b. Delete only the offenders: 'oc exec -c elasticsearch -n $NS $pod -- es_util --query=".kibana.*/_delete_by_query?pretty" -d "{\"query\":{\"match\":{\"title\":\"*TITLE*\"}}}"'

Comment 3 openshift-github-bot 2018-12-14 16:57:51 UTC
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/b51ce0f1de4a004136d9de36fcc96c11521ddd13
bug 1656086. remove index-patterns with title=

https://github.com/openshift/origin-aggregated-logging/commit/b362507e0888d799c8689b2875fb275dcc10b8ba
Merge pull request #1484 from openshift-cherrypick-robot/cherry-pick-1482-to-master

[master] bug 1656086. remove index-patterns with title=$TITLE

Comment 4 openshift-github-bot 2018-12-14 18:22:35 UTC
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/710f6e766f5ec4960e21b8131117b7527810bc8f
bug 1656086. code review fixes

https://github.com/openshift/origin-aggregated-logging/commit/452005329a41c68565a17f88a0f9f0938dcd42f9
Merge pull request #1487 from openshift-cherrypick-robot/cherry-pick-1485-to-master

[master] bug 1656086. code review fixes

Comment 6 Anping Li 2018-12-21 06:44:06 UTC
Reproduced with ose-logging-elasticsearch5:v3.11.51-3.The issue are fixed after upgraded to openshift3/ose-logging-elasticsearch5:v3.11.59.
Before v3.11.59, the Kibana index docs.count is 2 
green  open   .kibana.39a89dd158c8e9747943b00f84be79fc33fbd405                     U5T4rM_zSN6rfC2sB62N4A   1   0          2            0       55kb           55kb
In v3.11.59, the kibana index docs.count is 3
green  open   .kibana.39a89dd158c8e9747943b00f84be79fc33fbd405                     U5T4rM_zSN6rfC2sB62N4A   1   0          3            0     54.8kb         54.8kb

Comment 8 errata-xmlrpc 2019-01-10 09:04:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0024


Note You need to log in before you can comment on or make changes to this bug.