Bug 1275278 - [intservice_public_91]can't access kibana console after EFK stack successfully deployed
[intservice_public_91]can't access kibana console after EFK stack successfull...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging (Show other bugs)
3.1.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Jeff Cantrill
chunchen
: UpcomingRelease
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-26 08:46 EDT by wyue
Modified: 2016-09-29 22:16 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-23 09:43:40 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
capture screen of login kibana login request (219.96 KB, image/png)
2015-11-02 04:25 EST, wyue
no flags Details

  None (edit)
Description wyue 2015-10-26 08:46:06 EDT
Description of problem:
after deployment of EFK, logging-kibana pod is running without error in its logs, kibana web console still can't be accessed


Version-Release number of selected component (if applicable):
oc v3.0.2.903-29-g49953d6
openshift v3.0.2.903-29-g49953d6
kubernetes v1.2.0-alpha.1-1107-g4c8e6f4

How reproducible:
always

Steps to Reproduce:
1.deploy EFK according this guide:
https://github.com/openshift/origin-aggregated-logging/tree/master/deployment
2.wait until all pods are ready
3.update /etc/hosts in local machine, then access: https://kibana.example.com from browser.

Actual results:
can't access web console

Expected results:
web console should successfully displayed

Additional info:
1.are the values of below parameters correct? 
[root@dhcp-128-16 ~]# oc edit dc logging-kibana
        - name: OAP_MASTER_URL
          value: https://openshift-142.lab.eng.nay.redhat.com:8443
        - name: OAP_PUBLIC_MASTER_URL
          value: https://openshift-142.lab.eng.nay.redhat.com:8443
2.if OAP_MASTER_URL is 'https://kubernetes.default.svc.cluster.local', then
I can get a login page when accessing https://kibana.example.com, but I don't know what's the user/password

3.[root@dhcp-128-16 ~]# oc rsh logging-kibana-5-pkm8a
bash-4.2$ curl localhost:5601
<!DOCTYPE html>
  <!--[if IE 8]>         <html class="no-js lt-ie9" lang="en"> <![endif]-->
  <!--[if gt IE 8]><!--> <html class="no-js" lang="en"> <!--<![endif]-->
  <head>
    <meta charset="utf-8">
Comment 2 Luke Meyer 2015-10-26 09:22:03 EDT
If you can login at the OS web console (https://openshift-142.lab.eng.nay.redhat.com:8443/console) then you should be able to use the same login at the kibana web console, and have access to the same projects the user does at the OS web console. This is OAuth2-based authentication so the standard system:admin certificate can't be used; you have to create a user that logs in via OAuth2.

I do need to add a troubleshooting section to the deployment document though as there are a number of ways for the administrator to end up without working login and no apparent error or approach to fix.
Comment 3 Dan Mace 2015-10-26 09:38:04 EDT
Reassigning component as the deployment process itself is working.
Comment 4 wyue 2015-10-26 22:40:25 EDT
(In reply to Luke Meyer from comment #2)
> If you can login at the OS web console
> (https://openshift-142.lab.eng.nay.redhat.com:8443/console) then you should
> be able to use the same login at the kibana web console, and have access to
> the same projects the user does at the OS web console. This is OAuth2-based
> authentication so the standard system:admin certificate can't be used; you
> have to create a user that logs in via OAuth2.
> 
> I do need to add a troubleshooting section to the deployment document though
> as there are a number of ways for the administrator to end up without
> working login and no apparent error or approach to fix.

I can login https://openshift-142.lab.eng.nay.redhat.com:8443/console with user wyue, but can't login with kibana web console with the same user.
wyue is project admin of project 'logging' and also has a cluster-admin role.

I will try this again today.
Comment 5 wyue 2015-10-26 22:45:01 EDT
any differences between this two parameters in dc:logging-kibana?

- name: OAP_MASTER_URL
  value: https://openshift-142.lab.eng.nay.redhat.com:8443
- name: OAP_PUBLIC_MASTER_URL
  value: https://openshift-142.lab.eng.nay.redhat.com:8443
Comment 6 Luke Meyer 2015-10-29 12:09:56 EDT
(In reply to wyue from comment #5)
> any differences between this two parameters in dc:logging-kibana?
> 
> - name: OAP_MASTER_URL

You shouldn't normally need to set this. It is the master URL to be used for access from the Kibana pod, should resolve internally, defaults to kubernetes.default...

> - name: OAP_PUBLIC_MASTER_URL

This should be specified; it is where you want your browser to land when logging in, so should be "public" at least in being exposed and resolving for your web browser.

Logging in assumes you have OAuth2 authentication set up. Typically for testing purposes you would do this with an htpasswd entry (which I'm not sure ansible does by default any more): https://docs.openshift.com/enterprise/3.0/admin_guide/configuring_authentication.html#HTPasswdPasswordIdentityProvider

If you're able to login at the OpenShift console but not Kibana, then please describe exactly what happens when you try.

By the way I am working on a troubleshooting section at https://github.com/sosiouxme/origin-aggregated-logging/blob/20151028-docs/deployment/README.md#troubleshooting which you may find helpful (this will eventually merge to master and the official documentation).
Comment 7 wyue 2015-11-02 04:21:40 EST
(In reply to Luke Meyer from comment #2)
> If you can login at the OS web console
> (https://openshift-142.lab.eng.nay.redhat.com:8443/console) then you should
> be able to use the same login at the kibana web console, and have access to
> the same projects the user does at the OS web console. This is OAuth2-based
> authentication so the standard system:admin certificate can't be used; you
> have to create a user that logs in via OAuth2.
> 
> I do need to add a troubleshooting section to the deployment document though
> as there are a number of ways for the administrator to end up without
> working login and no apparent error or approach to fix.

added a pictures of login request, hope this can help.
what other logs do you need?I found no error in pod logs or describe pod
Comment 8 wyue 2015-11-02 04:25 EST
Created attachment 1088496 [details]
capture screen of login kibana login request
Comment 9 ewolinet 2015-11-02 10:02:16 EST
Are you able to login at the OpenShift console but not Kibana? If so, please describe exactly what happens when you try.

Is this still an issue given that you can log in per Bug 1277052?
Comment 10 wyue 2015-11-02 21:55:20 EST
(In reply to ewolinet from comment #9)
> Are you able to login at the OpenShift console but not Kibana? If so, please
> describe exactly what happens when you try.
> 
> Is this still an issue given that you can log in per Bug 1277052?

I think I already described that in previous comments and bug 1277052.
If I can login Kibana depends on if I update OAP_MASTER_URL in dc:logging-kibana.
1. after updating OAP_MASTER_URL:
I can access https://kibana.example.com without entering user/password and got bug 1277052.
2. without updating OAP_MASTER_URL:
when access https://kibana.example.com, it redirects to a login page and I can't login with project admin,but I can login Openshift web console with project admin.
So I report this bug.

There is no error in pod logs or describe pod, not sure what other info do you need?
Comment 11 Jeff Cantrill 2015-11-04 09:14:48 EST
i have seen a similiar issue when the oauthclient secret utilized by the kibana proxy does not match that of the server and/or if the oauthclient does not include the public url of kibana the deployment.  It will just cycle back to the login.  The deployer process uses the following to create the oauthclient:

    apiVersion: v1
    kind: OAuthClient
    metadata:
      name: kibana-proxy
    secret: ${OAUTH_SECRET}
    redirectURIs:
    - https://${KIBANA_HOSTNAME}
    - https://${KIBANA_OPS_HOSTNAME}

Please very the proxy is using the secret defined here and this entry includes the kibana host name.
Comment 13 Jeff Cantrill 2015-11-05 14:54:57 EST
Please try again with the latest images.  We found issues the the previous kibana image that would prevent successful login
Comment 15 chunchen 2015-11-06 02:20:01 EST
Tried the latest images, it still requires user and password when login the kibana web console. Is it correct?

[root@openshift-149 ~]# docker images|grep logging | grep rcm
rcm-img-docker01.build.eng.bos.redhat.com:5001/openshift3/logging-elasticsearch                  latest              7707f0d3f875        9 hours ago         394.4 MB
rcm-img-docker01.build.eng.bos.redhat.com:5001/openshift3/logging-kibana                         latest              305d78ea6219        14 hours ago        219.1 MB
Comment 16 chunchen 2015-11-06 02:55:09 EST
But do not need user and password when redirect to kibana console via click the "View Archive" button on OpenShift web console.
Comment 17 Jeff Cantrill 2015-11-06 08:15:05 EST
Requiring username/password should depend on the order in which you arrive at Kibana.  If you link from the console, you would already be auth'd and should not require an additional log in since the console passes your token to Kibana.  You should be able to then return to Kibana from either the console or directly and not need to auth since our Kibana plugin stashes your token in a session.  Directly visiting the console without without any kind of auth, I would expect you to be asked for a username/passowrd.
Comment 18 Jeff Cantrill 2015-11-06 11:24:47 EST
Believe we have resolved the blocking aspects of this original issue and that login needs to be verified based on comment 17
Comment 19 chunchen 2015-11-09 05:19:02 EST
(In reply to Jeff Cantrill from comment #17)
> Requiring username/password should depend on the order in which you arrive
> at Kibana.  If you link from the console, you would already be auth'd and
> should not require an additional log in since the console passes your token
> to Kibana.  You should be able to then return to Kibana from either the
> console or directly and not need to auth since our Kibana plugin stashes
> your token in a session.  Directly visiting the console without without any
> kind of auth, I would expect you to be asked for a username/passowrd.


Yes, as you said the kibana will ask for a username/passowrd when directly visiting the console and will not need to require an additional log in when accessing openshift web console firstly.
Comment 20 Jeff Cantrill 2015-11-09 08:41:36 EST
What are the remaining issues?  It appears you have confirmed my expectations so I would expect this issue to be able to be closed.
Comment 21 chunchen 2015-11-10 00:39:58 EST
No remaining issue for now, please help to change the bug's status to ON_QA, then I will mark it as verified.

Thanks!
Comment 22 chunchen 2015-11-10 20:56:22 EST
According to Comment #15 ~ 21, mark it as verified.

Note You need to log in before you can comment on or make changes to this bug.