Bug 1644008

Description of problem:
Customer in secured environment is seeing a few strange behaviors in the EFK stack.

In the non-ops kibana, they either get:

Courier Fetch: unhandled courier request error: [security_exception] no permissions for indices:data/read/mget 

Or a gateway timeout with similar/same message as:
https://github.com/elastic/kibana/issues/12707

These seem relatively unpredictable when it will occur (i.e. sometimes they'll get one message, then after clearing browser cache or using a different browser/client they'll get the other, etc)
  - They tried increasing timeouts as per unverified kcs https://access.redhat.com/solutions/3508851 but this didn't do much
     oc set env dc/logging-kibana ELASTICSEARCH_REQUESTTIMEOUT=119000 ELASTICSEARCH_SHARDTIMEOUT=119000
  - We considered that Elasticsearch might be too slow to respond -- INSTANCE_RAM and the limit are set to 6gb in one environment and 8gb in another. However, the cluster is essentially empty (only a single app running for test purposes, a single master, only a couple nodes). All the indices are green and healthy, fluentd is not reporting any errors. Also the issue occurs even when trying to view pages like the settings page, so it's not only when contacting the backend (so far as we can tell)

In the ops kibana, after authentication step, kibana replies to the client with just an empty response body. From the browser's perspective, it's an empty body but a 200 response message. The kibana logs seem to indicate the same -- a response length of 12 (maybe header?) and a 200.

They set up graphical tools on the master host so they could use a browser *from* the master -- same behavior.


Version-Release number of selected component (if applicable):
3.9

kibana 4.064
build 10229


How reproducible:
Unconfirmed. This affects multiple environments, and multiple image versions (they recently updated to the current latest 3.9 images just to be sure) but we can't seem to reproduce in test environments. I normally would suspect an environment or config issue, however since even kibana's logs seem to indicate a near-empty response body I'm not entirely sure.


Notes:
I figure we're missing something else reasonably "obvious" to look at, at least with respect to the timeouts -- but I'm not sure how that would affect the settings menu or etc.

Consulting is on site so while we can't get data out (logs, configs, etc) we can interface through them.