Bug 1870063 - [OCP 4.5] kibana index cannot be created after upgrade
Summary: [OCP 4.5] kibana index cannot be created after upgrade
Keywords:
Status: CLOSED DUPLICATE of bug 1867461
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.5
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: 4.6.0
Assignee: Periklis Tsirakidis
QA Contact: Anping Li
URL:
Whiteboard: logging-exploration
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-19 09:46 UTC by Mario Abajo
Modified: 2020-11-05 16:14 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-31 14:18:38 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Mario Abajo 2020-08-19 09:46:51 UTC
Description of problem:
After upgrade from 4.4 to 4.5 kibana indexes are not created and logs about the user shows this:

[2020-08-17T06:22:24,224][INFO ][c.a.o.s.p.PrivilegesEvaluator] [elasticsearch-cdm-g7ojzjry-2] No index-level perm match for User [name=Dirk.Egert, roles=[project_user], requestedTenant=__user__] Resolved [aliases=[.kibana_1687154965_dirkegert], indices=[], allIndices=[.kibana_1687154965_dirkegert_2], types=[doc], originalRequested=[.kibana_1687154965_dirkegert], remoteIndices=[]] [Action [indices:data/write/bulk[s]]] [RolesChecked [project_user]]
[2020-08-17T06:22:24,224][INFO ][c.a.o.s.p.PrivilegesEvaluator] [elasticsearch-cdm-g7ojzjry-2] No permissions for [indices:data/write/index, indices:data/write/bulk[s]]

i see that the index begin created/accessed is this one: 

.kibana_1687154965_dirkegert_2

i verified the roles extracting them with: 
$ oc exec <logging-es-POD> -- es_acl get --doc=roles

~~~
  "project_user": {
    "indices": {
...
      "?kibana_*_${user_name}": {
        "*": [
          "CRUD"
        ]
      }
    }, 
~~~

and find that this cannot match. I get a dump of the logging project and found that there is an alias created:

~~~
alias                        index                                                                 filter routing.index routing.search
.kibana_1687154965_dirkegert .kibana_1687154965_dirkegert_2                                        -      -             -
~~~

I would to verify if my analysis is correct. And if it is correct then look for answer to some questions:

1. Why this alias is created? is created as part of the upgrade? or was already there before?

2. how to resolve this situation? is it safe to just delete the alias?

I'm attaching privately the logs.

Thanks in advance

Comment 8 Jeff Cantrill 2020-08-28 01:30:17 UTC
(In reply to Mario Abajo from comment #0)
> Description of problem:
> After upgrade from 4.4 to 4.5 kibana indexes are not created and logs about
> the user shows this:
> 
> [2020-08-17T06:22:24,224][INFO ][c.a.o.s.p.PrivilegesEvaluator]
> [elasticsearch-cdm-g7ojzjry-2] No index-level perm match for User
> [name=Dirk.Egert, roles=[project_user], requestedTenant=__user__] 

Might the customer be able to test a username without a dot? I'm wondering if ${user_name} in the permission is not being evaluated correctly.

Comment 9 Periklis Tsirakidis 2020-08-28 13:08:03 UTC
@jeff

I can confirm this is related to the dot in the username. I've created a test-user `mig.user` and the index created is without the dot:

green open .kibana_-1613711986_miguser 5toXw2KlSZu5bYA60r1i0Q 3 1      1 0  8.4kb   4.2kb

Comment 10 Periklis Tsirakidis 2020-08-28 13:19:26 UTC
This seems to be a bug in OpenDistro: https://github.com/opendistro-for-elasticsearch/security/issues/74

Comment 11 Jeff Cantrill 2020-08-28 13:38:02 UTC
We can solve this issue without the OpenDistro fix by:

* add a modified user name as an alternate proxy var in ES proxy stripping out the dot
* Update the permissions in OAL to use the new var

Comment 12 Jeff Cantrill 2020-08-28 13:39:39 UTC
(In reply to Periklis Tsirakidis from comment #9)
> @jeff
> 
> I can confirm this is related to the dot in the username. I've created a
> test-user `mig.user` and the index created is without the dot:
> 
> green open .kibana_-1613711986_miguser 5toXw2KlSZu5bYA60r1i0Q 3 1      1 0 
> 8.4kb   4.2kb

And you confirm that you are unable to access kibana?

Comment 13 Periklis Tsirakidis 2020-08-31 14:18:38 UTC
Test results conclude that the dot in the username is not an issue neither for Kibana nor for OpenDistro Security. This is a duplicate of 1867461

*** This bug has been marked as a duplicate of bug 1867461 ***


Note You need to log in before you can comment on or make changes to this bug.