> [2020-11-26T03:30:42,749][INFO ][c.a.o.s.p.PrivilegesEvaluator] [elasticsearch-cdm-9aeuzkeh-1] No index-level perm match for User [name=@github, roles=[project_user], requestedTenant=__user__] Resolved [aliases=[.kibana_2099475215_github], indices=[], allIndices=[.kibana_2099475215_github_2], types=[doc], originalRequested=[.kibana_2099475215_github], remoteIndices=[]] [Action [indices:data/write/bulk[s]]] [RolesChecked [project_user]]

> [2020-11-26T03:31:57,992][INFO ][c.a.o.s.p.PrivilegesEvaluator] [elasticsearch-cdm-9aeuzkeh-1] No index-level perm match for User [name=@github, roles=[project_user], requestedTenant=__user__] Resolved [aliases=[.kibana_2099475215_github], indices=[], allIndices=[.kibana_2099475215_github_2], types=[doc], originalRequested=[.kibana_2099475215_github], remoteIndices=[]] [Action [indices:data/write/bulk[s]]] [RolesChecked [project_user]]

> [2020-11-26T03:29:31,499][INFO ][c.a.o.s.p.PrivilegesEvaluator] [elasticsearch-cdm-9aeuzkeh-3] No index-level perm match for User [name=github, roles=[project_user], requestedTenant=__user__] Resolved [aliases=[.kibana_2099475215_github], indices=[], allIndices=[.kibana_2099475215_github_2], types=[doc], originalRequested=[.kibana_2099475215_github], remoteIndices=[]] [Action [indices:data/write/bulk[s]]] [RolesChecked [project_user]]

These messages indicate that the user does not have access to the indices they are trying to access.
What is the index pattern that the user is trying to access their logs with?
Per our docs they should be one of "app", "infra" or "audit". Given this user is using the __user__ tenant, I would expect they should be using the "app" alias and index pattern.

They should only be / have access to their own projects logs. The environment is highly multi-tenant, so there are lots of users with access to a small percentage of total projects.

They should be using the app alias / pattern.

On a side note, we saw kibana go into a yellow state (like in BZ1913952) on a brand new OCP 4.5 cluster so it seems like it is still trying to do some kind of index migrations even on fresh 4.5+ clusters.

This seems to occur on a semi-regular basis for different users on 4.5. Is there anything else we can collect or that you would want to see or know from the specific user?

Matt

(In reply to Matthew Robson from comment #5)
> They should only be / have access to their own projects logs. The
> environment is highly multi-tenant, so there are lots of users with access
> to a small percentage of total projects.
> 
> They should be using the app alias / pattern.
> 
> On a side note, we saw kibana go into a yellow state (like in BZ1913952) on
> a brand new OCP 4.5 cluster so it seems like it is still trying to do some
> kind of index migrations even on fresh 4.5+ clusters.
> 
> This seems to occur on a semi-regular basis for different users on 4.5. Is
> there anything else we can collect or that you would want to see or know
> from the specific user?


This is controlled by the Kibana multitenancy plugins and should only be migrating when it finds there is something to migrate (the plugin makes that decision based on Kibana code).
If this is happening frequently that makes me think Kibana is restarting often, is this accurate?


I'm not sure what the user index files that you attached are... do you have a must gather for this cluster?

I can't tell if kibana has restarted at all in the must gather provided, it looks like everything is still running without restarts.

I do see in kibana that the elasticsearch plugin was red and looking in the elasticsearch logs there appears to be an issue with the customer's application data because ES is failing to execute a bulk create for index app-000048.


2020-11-21T08:00:18.65170917Z [2020-11-21T08:00:18,651][DEBUG][o.e.a.b.TransportShardBulkAction] [elasticsearch-cdm-9aeuzkeh-1] [app-000048][1] failed to execute bulk item (create) index {[app-write][_doc][NjIwMTczMTUtNThmNy00ZTRiLTk4MzgtYzRjNWMwNWVjNDhj], source[n/a, actual length: [2kb], max length: 2kb]}
2020-11-21T08:00:18.65170917Z org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [message] of type [text] in document with id 'NjIwMTczMTUtNThmNy00ZTRiLTk4MzgtYzRjNWMwNWVjNDhj'

There appears to be utf formatting issues based on another ES node:

2020-11-21T08:00:19.873249955Z Caused by: com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0x8b


The fact that kibana cannot talk to ES may be the reason the plugin is yellow, however, looking at the current must gather it is unclear why kibana cannot talk to elasticsearch.

Is the customer still seeing the issue that was originally reported?
It was noted that only two customers are unable to use kibana, is it possible that their logs have not yet made it into Elasticsearch so that is preventing them from having the correct permissions?

Do you have an admin user that can check/confirm this?

2020-11-23T18:04:54.416956165Z [2020-11-23T18:04:54,416][INFO ][c.a.o.s.p.PrivilegesEvaluator] [elasticsearch-cdm-9aeuzkeh-2] No index-level perm match for User [name=<redacted>@github, roles=[project_user], requestedTenant=__user__] Resolved [aliases=[*], indices=[*], allIndices=[*], types=[*], originalRequested=[], remoteIndices=[]] [Action [indices:admin/get]] [RolesChecked [project_user]]
2020-11-23T18:04:54.417041035Z [2020-11-23T18:04:54,417][INFO ][c.a.o.s.p.PrivilegesEvaluator] [elasticsearch-cdm-9aeuzkeh-2] No permissions for [indices:admin/get]

From what I know, Kibana is not restarting often, but I can't say for sure if there was a restart around the time of the issue. I will note that as something to check when it happens again.

The 'failed to execute bulk item' could be coincidental, aside from these one off index issues, the logging stack is generally operational and when they do see this issue, deleting the users kibana index results in immediate resolution to the problem. In this specific case, Kibana is not yellow (that's https://bugzilla.redhat.com/show_bug.cgi?id=1913952 in which Kiabna doesn't work for anyone). This is generally a single user issue where Kibana works fine for everyone else, admins or otherwise.

This issue, anecdotally, looks to occur more often within a group of people who use Kibana significantly more often.

A bunch of users who have experienced this have had it happen more than once. They all have long standing projects with lots of logs. It's possible they had a new project created recently, but I don't know how we could correlate that happening and the impact.

When this happens again for a user, is there any more info you would want to collect outside a new must gather? I can grab another must gather to compare the circumstances to the initial observations.

*** This bug has been marked as a duplicate of bug 1933978 ***