Bug 1734128 - Kibana 500 error, trying to resolve back to the client that is requesting the kibana UI [NEEDINFO]
Summary: Kibana 500 error, trying to resolve back to the client that is requesting the...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.11.z
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-29 16:57 UTC by Dirk Porter
Modified: 2019-10-24 12:47 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-24 12:47:14 UTC
Target Upstream Version:
jcantril: needinfo? (dporter)
jcantril: needinfo? (dporter)


Attachments (Terms of Use)
Kibana-proxy logs (22.06 KB, text/plain)
2019-07-29 16:57 UTC, Dirk Porter
no flags Details

Description Dirk Porter 2019-07-29 16:57:15 UTC
Created attachment 1594379 [details]
Kibana-proxy logs

Description of problem:
The customer has a cluster deployed in Azure with a LB forwarding requests from port 4401 to 443. The customer has performed the following to redirect properly when requesting the site: 

oc edit dc logging-kibana 

        - -redirect-url=https://kibana.vycld-preprod-us.dieboldservices.local:4401/oauth2/callback
        - -redeem-url=https://kubernetes.default.svc/oauth/token
        - -login-url=https://openshift.vycld-preprod-us.dieboldservices.local:4400/oauth/authorize
        - -request-logging=true

The redirections are working correctly in terms of the correct URL being utilized, however the following message appears after logging in: 

message	"An internal server error occurred"
statusCode	500
error	"Internal Server Error"

Additionally, it appears it is failing due to openshift attempting to resolve the customer's PC ip address(10.39.137.125:50594) as seen below: 

{"type":"response","@timestamp":"2019-07-26T15:58:11Z","tags":[],"pid":228,"method":"get","statusCode":200,"req":{"url":"/","method":"get","headers":{"host":"kibana.vycld-preprod-us.dieboldservices.local","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.5","dnt":"1","forwarded":"for=10.176.135.136;host=kibana.vycld-preprod-us.dieboldservices.local;proto=https;proto-version=","referer":"https://openshift.vycld-preprod-us.dieboldservices.local:4400/","upgrade-insecure-requests":"1","x-forwarded-access-token":"Wsh-DCCh9iqgWdRcMOPwdOpo4IAA0_K6nhPEqoCPZUs","x-forwarded-email":"mauricio . (9F7C228C)@cluster.local","x-forwarded-for":"10.39.137.125:50594, 10.176.135.136, 10.1.3.1","x-forwarded-host":"kibana.vycld-preprod-us.dieboldservices.local","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-user":"mauricio. (9F7C228C)","x-original-host":"kibana.vycld-preprod-us.dieboldservices.local:4401","x-original-url":"/"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://openshift.vycld-preprod-us.dieboldservices.local:4400/"},"res":{"statusCode":200,"responseTime":9,"contentLength":9},"message":"GET  / 200 9ms - 9.0B"}


Version-Release number of selected component (if applicable)


Steps to Reproduce:
1. Port Forward on an external facing lB in this instance 4401 --> 443 
2. Edit the deployment config for logging-kibana(redirect-url, redeem-url, login-url) 
3. Try to log into kibana on the forwarded port.

Actual results:

500 Error

Expected results:

Expected to see the kibana UI 

Additional info:

Comment 2 Dirk Porter 2019-07-29 20:30:35 UTC
I did not see any in the logs that I uploaded. Would you like me to try to get more verbose logs?

Comment 3 Dirk Porter 2019-08-02 17:02:42 UTC
Hello, 

Did you need any further information? 

Regards, 

Dirk Porter

Comment 7 Jeff Cantrill 2019-08-21 20:24:28 UTC
Please check to see if the secrets are still in sync:

test "$(oc get secret logging-kibana-proxy -o jsonpath={.data.oauth-secret} | base64 -d)" = "$(oc get oauthclient kibana-proxy -o jsonpath={.secret})";echo $?

Comment 8 Jeff Cantrill 2019-08-22 12:18:18 UTC
One additional thing I thought of was did you also edit the oauthclient to modify the redirect-url?  Example of mine:

# oc get oauthclient kibana-proxy -o yaml
accessTokenMaxAgeSeconds: 604800
apiVersion: oauth.openshift.io/v1
kind: OAuthClient
metadata:
  labels:
    logging-infra: support
  name: kibana-proxy
redirectURIs:
- https://kibana.192.168.100.212.nip.io
scopeRestrictions:
- literals:
  - user:info
  - user:check-access
  - user:list-projects
secret: zNhb9FbtXFPPlnAR9ccshPF41tBKBUFjXxF4VIXFA9uyVJnX6ODEFHIakOGsmMy2

Upon further investigation, we don't set any of those options for the proxy container [1] so how it may behave is untested.


Additionally note this change will be reverted by ansible upon any upgrade.

[1] https://github.com/openshift/openshift-ansible/blob/release-3.11/roles/openshift_logging_kibana/templates/kibana.j2#L99-L110

Comment 9 Jeff Cantrill 2019-10-24 12:47:14 UTC
Closing since the customer case was closed.


Note You need to log in before you can comment on or make changes to this bug.