Bug 1883673

Summary: Error 500: Internal Server Error on Kibana
Product: OpenShift Container Platform Reporter: Jeff Cantrill <jcantril>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: high Docs Contact:
Priority: high    
Version: 4.5CC: antgarci, aos-bugs, bfurtado, cpassare, dkulkarn, ewolinet, halim.lee, hfukumot, jaliang, jcantril, jmalde, joboyer, jpessott, mabajodu, mas-hatada, mharri, naygupta, oarribas, ocasalsa, openshift-bugs-escalate, puraut, qitang, rh-container, rsandu, ychoukse, ykarajag
Target Milestone: ---   
Target Release: 4.5.z   
Hardware: Unspecified   
OS: Mac OS   
Whiteboard: logging-exploration
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1866490 Environment:
Last Closed: 2020-10-26 14:55:25 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1866490    
Bug Blocks: 1723620, 1878029, 1878305    

Description Jeff Cantrill 2020-09-29 20:08:09 UTC
+++ This bug was initially created as a clone of Bug #1866490 +++

Description of problem:

When I set up the cluster logging instance (I've attached a yaml file for configuration), all pods start and run successfully.
Then after a few days, I encounter an error message when I open Kibana route:
"{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}"

Version-Release number of selected component (if applicable):
Cluster Logging 4.5.0-202007012112

How reproducible:
A few days after setting up cluster logging instance

Steps to Reproduce:
1. Install Cluster Logging operator
2. Setting up Cluster Logging instance
3. After a few days, Kibana will throw Error 500

Actual results:
Kibana log:
  {"type":"error","@timestamp":"2020-07-29T15:00:27Z","tags":[],"pid":121,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Internal Server Error :: {\"path\":\"/.kibana/doc/config%3A6.8.1\",\"query\":{},\"statusCode\":500,\"response\":\"{\\\"code\\\":500,\\\"message\\\":\\\"Internal Error\\\",\\\"error\\\":{}}\\n\"}\n    at respond (/opt/app-root/src/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/opt/app-root/src/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/opt/app-root/src/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n    at IncomingMessage.wrapper (/opt/app-root/src/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/app/kibana","path":"/app/kibana","href":"/app/kibana"},"message":"Internal Server Error"}

Expected results:
Kibana page loading properly

Additional info:
I could successfully access through Google Chrome incognito mode, while the usual chrome browser still experienced error 500. It looks like authentication process is related.

Comment 1 Jeff Cantrill 2020-09-29 21:04:06 UTC
*** Bug 1872104 has been marked as a duplicate of this bug. ***

Comment 5 Anping Li 2020-10-20 06:10:48 UTC
Verfied on clusterlogging.4.5.0-202010161522.p0,elasticsearch-operator.4.5.0-202010161522.p0.

Comment 7 errata-xmlrpc 2020-10-26 14:55:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5.16 extras update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4270

Comment 8 Jace Liang 2021-08-06 06:03:57 UTC
IHAC In OpenShift 4.5.30 having this issue.
Using Chrome in incognito mode can solve the problem.
But most of the time they need to close all the Chrome windows (including normal and incognito mode) to access kibana successfully.

Once they create a support case for this, I will update the case ID here.

Comment 9 Jace Liang 2021-08-09 06:38:02 UTC
Update the related case: #03005937

https://gss--c.visualforce.com/apex/Case_View?id=5002K00000xDG2R&sfdc.override=1