Bug 1870063

Summary: [OCP 4.5] kibana index cannot be created after upgrade
Product: OpenShift Container Platform Reporter: Mario Abajo <mabajodu>
Component: LoggingAssignee: Periklis Tsirakidis <periklis>
Status: CLOSED DUPLICATE QA Contact: Anping Li <anli>
Severity: medium Docs Contact:
Priority: high    
Version: 4.5CC: aconway, aos-bugs, jcantril, periklis
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: logging-exploration
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-31 14:18:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mario Abajo 2020-08-19 09:46:51 UTC
Description of problem:
After upgrade from 4.4 to 4.5 kibana indexes are not created and logs about the user shows this:

[2020-08-17T06:22:24,224][INFO ][c.a.o.s.p.PrivilegesEvaluator] [elasticsearch-cdm-g7ojzjry-2] No index-level perm match for User [name=Dirk.Egert, roles=[project_user], requestedTenant=__user__] Resolved [aliases=[.kibana_1687154965_dirkegert], indices=[], allIndices=[.kibana_1687154965_dirkegert_2], types=[doc], originalRequested=[.kibana_1687154965_dirkegert], remoteIndices=[]] [Action [indices:data/write/bulk[s]]] [RolesChecked [project_user]]
[2020-08-17T06:22:24,224][INFO ][c.a.o.s.p.PrivilegesEvaluator] [elasticsearch-cdm-g7ojzjry-2] No permissions for [indices:data/write/index, indices:data/write/bulk[s]]

i see that the index begin created/accessed is this one: 

.kibana_1687154965_dirkegert_2

i verified the roles extracting them with: 
$ oc exec <logging-es-POD> -- es_acl get --doc=roles

~~~
  "project_user": {
    "indices": {
...
      "?kibana_*_${user_name}": {
        "*": [
          "CRUD"
        ]
      }
    }, 
~~~

and find that this cannot match. I get a dump of the logging project and found that there is an alias created:

~~~
alias                        index                                                                 filter routing.index routing.search
.kibana_1687154965_dirkegert .kibana_1687154965_dirkegert_2                                        -      -             -
~~~

I would to verify if my analysis is correct. And if it is correct then look for answer to some questions:

1. Why this alias is created? is created as part of the upgrade? or was already there before?

2. how to resolve this situation? is it safe to just delete the alias?

I'm attaching privately the logs.

Thanks in advance

Comment 8 Jeff Cantrill 2020-08-28 01:30:17 UTC
(In reply to Mario Abajo from comment #0)
> Description of problem:
> After upgrade from 4.4 to 4.5 kibana indexes are not created and logs about
> the user shows this:
> 
> [2020-08-17T06:22:24,224][INFO ][c.a.o.s.p.PrivilegesEvaluator]
> [elasticsearch-cdm-g7ojzjry-2] No index-level perm match for User
> [name=Dirk.Egert, roles=[project_user], requestedTenant=__user__] 

Might the customer be able to test a username without a dot? I'm wondering if ${user_name} in the permission is not being evaluated correctly.

Comment 9 Periklis Tsirakidis 2020-08-28 13:08:03 UTC
@jeff

I can confirm this is related to the dot in the username. I've created a test-user `mig.user` and the index created is without the dot:

green open .kibana_-1613711986_miguser 5toXw2KlSZu5bYA60r1i0Q 3 1      1 0  8.4kb   4.2kb

Comment 10 Periklis Tsirakidis 2020-08-28 13:19:26 UTC
This seems to be a bug in OpenDistro: https://github.com/opendistro-for-elasticsearch/security/issues/74

Comment 11 Jeff Cantrill 2020-08-28 13:38:02 UTC
We can solve this issue without the OpenDistro fix by:

* add a modified user name as an alternate proxy var in ES proxy stripping out the dot
* Update the permissions in OAL to use the new var

Comment 12 Jeff Cantrill 2020-08-28 13:39:39 UTC
(In reply to Periklis Tsirakidis from comment #9)
> @jeff
> 
> I can confirm this is related to the dot in the username. I've created a
> test-user `mig.user` and the index created is without the dot:
> 
> green open .kibana_-1613711986_miguser 5toXw2KlSZu5bYA60r1i0Q 3 1      1 0 
> 8.4kb   4.2kb

And you confirm that you are unable to access kibana?

Comment 13 Periklis Tsirakidis 2020-08-31 14:18:38 UTC
Test results conclude that the dot in the username is not an issue neither for Kibana nor for OpenDistro Security. This is a duplicate of 1867461

*** This bug has been marked as a duplicate of bug 1867461 ***