Bug 1734128 - Kibana 500 error, trying to resolve back to the client that is requesting the kibana UI [NEEDINFO]
Summary: Kibana 500 error, trying to resolve back to the client that is requesting the...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.11.z
Assignee: Jeff Cantrill
QA Contact: Anping Li
Depends On:
TreeView+ depends on / blocked
Reported: 2019-07-29 16:57 UTC by Dirk Porter
Modified: 2019-10-24 12:47 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-10-24 12:47:14 UTC
Target Upstream Version:
jcantril: needinfo? (dporter)
jcantril: needinfo? (dporter)

Attachments (Terms of Use)
Kibana-proxy logs (22.06 KB, text/plain)
2019-07-29 16:57 UTC, Dirk Porter
no flags Details

Description Dirk Porter 2019-07-29 16:57:15 UTC
Created attachment 1594379 [details]
Kibana-proxy logs

Description of problem:
The customer has a cluster deployed in Azure with a LB forwarding requests from port 4401 to 443. The customer has performed the following to redirect properly when requesting the site: 

oc edit dc logging-kibana 

        - -redirect-url=https://kibana.vycld-preprod-us.dieboldservices.local:4401/oauth2/callback
        - -redeem-url=https://kubernetes.default.svc/oauth/token
        - -login-url=https://openshift.vycld-preprod-us.dieboldservices.local:4400/oauth/authorize
        - -request-logging=true

The redirections are working correctly in terms of the correct URL being utilized, however the following message appears after logging in: 

message	"An internal server error occurred"
statusCode	500
error	"Internal Server Error"

Additionally, it appears it is failing due to openshift attempting to resolve the customer's PC ip address( as seen below: 

{"type":"response","@timestamp":"2019-07-26T15:58:11Z","tags":[],"pid":228,"method":"get","statusCode":200,"req":{"url":"/","method":"get","headers":{"host":"kibana.vycld-preprod-us.dieboldservices.local","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.5","dnt":"1","forwarded":"for=;host=kibana.vycld-preprod-us.dieboldservices.local;proto=https;proto-version=","referer":"https://openshift.vycld-preprod-us.dieboldservices.local:4400/","upgrade-insecure-requests":"1","x-forwarded-access-token":"Wsh-DCCh9iqgWdRcMOPwdOpo4IAA0_K6nhPEqoCPZUs","x-forwarded-email":"mauricio . (9F7C228C)@cluster.local","x-forwarded-for":",,","x-forwarded-host":"kibana.vycld-preprod-us.dieboldservices.local","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-user":"mauricio. (9F7C228C)","x-original-host":"kibana.vycld-preprod-us.dieboldservices.local:4401","x-original-url":"/"},"remoteAddress":"","userAgent":"","referer":"https://openshift.vycld-preprod-us.dieboldservices.local:4400/"},"res":{"statusCode":200,"responseTime":9,"contentLength":9},"message":"GET  / 200 9ms - 9.0B"}

Version-Release number of selected component (if applicable)

Steps to Reproduce:
1. Port Forward on an external facing lB in this instance 4401 --> 443 
2. Edit the deployment config for logging-kibana(redirect-url, redeem-url, login-url) 
3. Try to log into kibana on the forwarded port.

Actual results:

500 Error

Expected results:

Expected to see the kibana UI 

Additional info:

Comment 2 Dirk Porter 2019-07-29 20:30:35 UTC
I did not see any in the logs that I uploaded. Would you like me to try to get more verbose logs?

Comment 3 Dirk Porter 2019-08-02 17:02:42 UTC

Did you need any further information? 


Dirk Porter

Comment 7 Jeff Cantrill 2019-08-21 20:24:28 UTC
Please check to see if the secrets are still in sync:

test "$(oc get secret logging-kibana-proxy -o jsonpath={.data.oauth-secret} | base64 -d)" = "$(oc get oauthclient kibana-proxy -o jsonpath={.secret})";echo $?

Comment 8 Jeff Cantrill 2019-08-22 12:18:18 UTC
One additional thing I thought of was did you also edit the oauthclient to modify the redirect-url?  Example of mine:

# oc get oauthclient kibana-proxy -o yaml
accessTokenMaxAgeSeconds: 604800
apiVersion: oauth.openshift.io/v1
kind: OAuthClient
    logging-infra: support
  name: kibana-proxy
- https://kibana.
- literals:
  - user:info
  - user:check-access
  - user:list-projects
secret: zNhb9FbtXFPPlnAR9ccshPF41tBKBUFjXxF4VIXFA9uyVJnX6ODEFHIakOGsmMy2

Upon further investigation, we don't set any of those options for the proxy container [1] so how it may behave is untested.

Additionally note this change will be reverted by ansible upon any upgrade.

[1] https://github.com/openshift/openshift-ansible/blob/release-3.11/roles/openshift_logging_kibana/templates/kibana.j2#L99-L110

Comment 9 Jeff Cantrill 2019-10-24 12:47:14 UTC
Closing since the customer case was closed.

Note You need to log in before you can comment on or make changes to this bug.