Description of problem: Kibana login returns fatal error upon login: ``` Fatal Error Courier Fetch Error: unhandled courier request error: [security_exception] no permissions for indices:data/read/mget Version: 4.6.4 Build: 10229 Error: unhandled courier request error: [security_exception] no permissions for indices:data/read/mget" ``` Version-Release number of selected component (if applicable): v3.10.34 Actual results: Kibana login return "Error: unhandled courier request error: [security_exception] no permissions for indices:data/read/mget" Expected results: Kibana login to work properly. Additional info: - Similar issue in [1], solved through an errata in v3.10.15. However, in this case, the customer has v3.10.34 deployed. - Indices seem healthy. - Doesn't seems permissions or retention time frame related: same error is reproduced by cluster-admin users upon login.
Reviewing the logs I see several instances of ES nodes dropping our of the ES cluster. * Can you check the connectivity to the ES nodes to verify the latency [2]? * Maybe also check the connectivity from Kibana [3] The gateway timeout can be mitigated by adjusting the request timeout [1] [1] https://github.com/openshift/origin-aggregated-logging/tree/release-3.10/kibana#configuration-modifications [2] https://github.com/jcantrill/cluster-logging-tools/blob/master/scripts/check-es-cluster-connectivity [3] https://github.com/jcantrill/cluster-logging-tools/blob/master/scripts/check-kibana-to-es-connectivity
Hi. Increasing ELASTICSEARCH_REQUESTTIMEOUT seems to make no difference. Customer is still seeing gateway timeout" errors when accessing Kibana dashboard. Attached you can find the outputs of the ES latency test and Kibana connectivity.
I believe this will be resolved by https://bugzilla.redhat.com/show_bug.cgi?id=1705589 and subsequent backports to 3.11. The upstream issues are merged and I am doing the work now to get them built for a production release.
(In reply to Jeff Cantrill from comment #13) > I believe this will be resolved by > https://bugzilla.redhat.com/show_bug.cgi?id=1705589 and subsequent backports > to 3.11. The upstream issues are merged and I am doing the work now to get > them built for a production release. Hi Jeff. Which 3.11 z-stream version has been (or will be) backported to? Thank you.
Closing as duplicate to https://bugzilla.redhat.com/show_bug.cgi?id=1705589 . Will backport to 3.11. Please reopen if not resolved *** This bug has been marked as a duplicate of bug 1705589 ***
Changing status to WONTFIX as we will resolve foe 3.11 but not 3.10