Bug 1705589 - kibana presenting blank page or timeout [NEEDINFO]
Summary: kibana presenting blank page or timeout
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 4.2.0
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On: 1644008
Blocks: 1726433 1726434
TreeView+ depends on / blocked
 
Reported: 2019-05-02 14:14 UTC by Josh Foots
Modified: 2019-10-16 06:28 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The dynamic seeding is inefficient and user's with visibility to a large number of projects exposes this issue by making too many calls to the Elasticsearch cluster. Additionally, there is no caching for user requests causing the seeding process to occur more frequently. Consequence: Calls to Elasticsearch can timeout before a response is returned. Fix: Introduce caching of the API calls and ACL seeding Result: Dynamic seeding is more efficient reducing the opportunity for page timeouts.
Clone Of: 1644008
: 1726433 1726434 (view as bug list)
Environment:
Last Closed: 2019-10-16 06:28:21 UTC
Target Upstream Version:
jcantril: needinfo? (jfoots)


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 1683 None closed Bug 1705589. Consume newer version of plugin to mitigate blank page 2020-11-06 12:40:27 UTC
Red Hat Product Errata RHBA-2019:2922 None None None 2019-10-16 06:28:33 UTC

Comment 3 Jeff Cantrill 2019-05-02 14:49:11 UTC
I did not see anything in the logs which would indicate permissions are not seeding properly.  Consider the following:

* Have they cleared their browser cache?  Does this problem still exist when using an incognito browser?
* Are there any errors on the browser side that can be posted here
* Is their high network latency between ES and Kibana which could be causing issues [1]
* Can you ask them to log in again and dump the permissions within 60s [2]

Any latency may require them to bump the request timeout [3] which we patched into Kibana 5 with [4]


[1] https://github.com/jcantrill/cluster-logging-tools/blob/master/scripts/check-kibana-to-es-connectivity
[2] https://github.com/jcantrill/cluster-logging-tools/blob/master/scripts/view-es-permissions
[3] https://github.com/openshift/origin-aggregated-logging/tree/master/kibana#configuration-modifications
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1679159

Comment 9 Jeff Cantrill 2019-05-06 19:46:07 UTC
Try the following:

1. rsh into one of the ES pods
2. change to the $HOME/sgconfig dir which should be a child to the one in which you are initially placed
3. Edit the sg_action_group.yaml file and add SEARCH to this group [1] 
4. Run 'es_seed_acl' - this will reseed the permissions

Note this is a work around and only temporary.  If any of the pods are restarted, you will need to re-execute these steps.

Also note the permission script from c#3 is only useful when run within 60s of the initial failure.  Permissions are expired after that time.


[1] https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/sgconfig/sg_action_groups.yml#L68

Comment 10 Jeff Cantrill 2019-05-06 21:23:13 UTC
Josh,

I am unable to reproduce locally.  Can you provide the all the steps interacting with Kibana? (e.g. log in, click discover, execute search, etc0

Comment 16 Jeff Cantrill 2019-05-09 12:24:42 UTC
Josh,

I dont see any response to my request for:

> Try the following:
> 
> 1. rsh into one of the ES pods
> 2. change to the $HOME/sgconfig dir which should be a child to the one in
> which you are initially placed
> 3. Edit the sg_action_group.yaml file and add SEARCH to this group [1] 
> 4. Run 'es_seed_acl' - this will reseed the permissions
> 
> Note this is a work around and only temporary.  If any of the pods are
> restarted, you will need to re-execute these steps.
> 
> Also note the permission script from c#3 is only useful when run within 60s
> of the initial failure.  Permissions are expired after that time.
> 
> 
> [1]
> https://github.com/openshift/origin-aggregated-logging/blob/master/
> elasticsearch/sgconfig/sg_action_groups.yml#L68

Comment 17 Jeff Cantrill 2019-07-02 20:15:26 UTC
*** Bug 1705026 has been marked as a duplicate of this bug. ***

Comment 19 Anping Li 2019-07-25 15:22:39 UTC
Couldn't reproduce this bug.  Move to verified as regression testing pass.

Comment 20 errata-xmlrpc 2019-10-16 06:28:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.