Description of problem: After an upgrade of OCP from 3.10 to 3.11, certain users were unable to view log data. The error being observed in the logs for those users were " Timelion: Error: in cell #1: [security_exception] no permissions for [indices:data/read/search] and User [name=<USERNAME>, roles=[gen_user_<UUID>, gen_kibana_<UUID>]] " Deleting the users index from the kibana index in elasticsearch using the procedure from https://access.redhat.com/solutions/3722751 then allows the user to view data again, however this procedure results in the user losing any custom queries they had stored. Version-Release number of selected component (if applicable): openshift-3.11.82 How reproducible: Occurred a number of times at customer, not been able to reproduce Steps to Reproduce: 1. 2. 3. Actual results: User is unable to access log data Expected results: User is able to access log data Additional info:
(In reply to Stuart Auchterlonie from comment #0) > > " > Timelion: Error: in cell #1: [security_exception] no permissions for > [indices:data/read/search] and User [name=<USERNAME>, > roles=[gen_user_<UUID>, gen_kibana_<UUID>]] > " > I don't believe we explicitly tested the timelion plugin so it very well may be that permissions don't exist to allow user's to see any index used to store timeline information. We only generate permissions for the indices: .operations, .orphaned, .kibana.<userhash>, project.<namespace>.<projectuid>.<date> Is it possible for you to identify what query is being executed against which index? > Deleting the users index from the kibana index in elasticsearch using > the procedure from https://access.redhat.com/solutions/3722751 > then allows the user to view data again, however this procedure > results in the user losing any custom queries they had stored. I'm not certain where this solution originated but it explicitly says 'Export any custom index-patterns, visualizations, dashboards' which addresses the loss of visualization. The solution does not, however, describe how to import them back to the index which could be accomplished with a curl statement or likely Kibana has and import function.
Looks like it might be possible to import dashboards in Kibana5 -> https://www.elastic.co/guide/en/kibana/5.0/loading-a-saved-dashboard.html I'll try it out and report back here
Ok, exporting / importing appears to work. Management -> Kibana -> Saved Objects -> Export Everything / Import. Works even once I delete my kibana index, refresh, import. I'll update customer as well.
This is not a bug. User's are only allowed to perform actions against indices for which you have permissions. If you are a cluster-admin then you can see all the data in the ES cluster. If you are a non-admin, you can only view data against the indices which are a 1:1 match to the Openshift namespaces you can access. I stood up a 3.11 cluster and loaded Timelion in Kibana. The initial load provides a text box that is trying to do something against all indices: .es(*) This is reporting correctly for non-admin users. If I modify it as advised in the ES documentation [1] to my test project: .es(index=project.myproject.5c466a17-8961-11e9-b2db-525400b6beac.*) which is in the form of project.<namespace>.<namespace_uuid>.<date> the result is a graph of the data available. [1] https://www.elastic.co/guide/en/kibana/5.6/timelion-create.html
Following up based on the following information from the client: * Kibana users before upgrade are unable to see their visualizations after upgrade without deleting their kibana index I am unable to reproduce by: * Installing latest 3.10 with kibana index mode unique * Log in as a cluster admin and create queries, visualizations, and dashboards * Upgrade the cluster to latest 3.11 * Log in as a cluster admin and load queries, visualizations, and dashboards Note there are no failures. This makes me think we are running into [1] which is related to load, specifically the number of projects associated with a particular user. User's with a large number of projects prior to [1] can run into timeout conditions where seeding is not complete and subsequently users would see a permissions denied issue. There is nothing in the code or the way Kibana loads these assets that should require deleting the Kibana index; I'm not certain from where that advise originated. Per the discussion with the customer, we will provide a script, as a work around that seems to mitigate the issue, to export and import a user's assets from their kibana index. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1726433
The function testing pass using logging:v3.11.135
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2352