Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1609895

Summary: Kibana 3.11.0-0.10.0 is trying to contact ES using localhost instead of logging-es - can't login to Kibana
Product: OpenShift Container Platform Reporter: Mike Fiedler <mifiedle>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.11.0CC: aos-bugs, juzhao, rmeggins
Target Milestone: ---Keywords: OnlineStarter
Target Release: 3.11.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Kibana was starting without using the kibana.yml we expected Consequence: Kibana was relying on its defaults instead of using our desired configuration Fix: Modify the run script to set the environment variable which tells Kibana which config to use Result: Kibana is able to connect to Elasticsearch
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-11 07:22:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mike Fiedler 2018-07-30 18:27:06 UTC 
Description of problem:

In 3.11, logging into Kibana gives 2 errors:

 Elasticsearch plugin is red
 Unable to connect to Elasticsearch at http://localhost:9200.

The kibana pod logs are full of:

{"type":"log","@timestamp":"2018-07-30T17:29:04Z","tags":["warning","elasticsearch","admin"],"pid":226,"message":"Unable to revive connection: http://localhost:9200/"}
{"type":"log","@timestamp":"2018-07-30T17:29:04Z","tags":["warning","elasticsearch","admin"],"pid":226,"message":"No living connections"}
{"type":"log","@timestamp":"2018-07-30T17:29:07Z","tags":["warning","elasticsearch","admin"],"pid":226,"message":"Unable to revive connection: http://localhost:9200/"}
{"type":"log","@timestamp":"2018-07-30T17:29:07Z","tags":["warning","elasticsearch","admin"],"pid":226,"message":"No living connections"}


The env vars all look ok - no use of localhost:

ELASTICSEARCH_URL=https://logging-es:9200
KIBANA_BIN=/usr/share/kibana/bin/kibana
KIBANA_CONF_DIR=/etc/kibana
KIBANA_HOME=/usr/share/kibana
KIBANA_MEMORY_LIMIT=771751936
KIBANA_VER=5.6.10
KUBERNETES_PORT=tcp://172.24.0.1:443
KUBERNETES_PORT_443_TCP=tcp://172.24.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=172.24.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_53_TCP=tcp://172.24.0.1:53
KUBERNETES_PORT_53_TCP_ADDR=172.24.0.1
KUBERNETES_PORT_53_TCP_PORT=53
KUBERNETES_PORT_53_TCP_PROTO=tcp
KUBERNETES_PORT_53_UDP=udp://172.24.0.1:53
KUBERNETES_PORT_53_UDP_ADDR=172.24.0.1
KUBERNETES_PORT_53_UDP_PORT=53
KUBERNETES_PORT_53_UDP_PROTO=udp
KUBERNETES_SERVICE_HOST=172.24.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_DNS=53
KUBERNETES_SERVICE_PORT_DNS_TCP=53
KUBERNETES_SERVICE_PORT_HTTPS=443
LOGGING_ES_PORT=tcp://172.26.233.225:9200
LOGGING_ES_PORT_9200_TCP=tcp://172.26.233.225:9200
LOGGING_ES_PORT_9200_TCP_ADDR=172.26.233.225
LOGGING_ES_PORT_9200_TCP_PORT=9200
LOGGING_ES_PORT_9200_TCP_PROTO=tcp
LOGGING_ES_PROMETHEUS_PORT=tcp://172.26.28.112:443
LOGGING_ES_PROMETHEUS_PORT_443_TCP=tcp://172.26.28.112:443
LOGGING_ES_PROMETHEUS_PORT_443_TCP_ADDR=172.26.28.112
LOGGING_ES_PROMETHEUS_PORT_443_TCP_PORT=443
LOGGING_ES_PROMETHEUS_PORT_443_TCP_PROTO=tcp
LOGGING_ES_PROMETHEUS_SERVICE_HOST=172.26.28.112
LOGGING_ES_PROMETHEUS_SERVICE_PORT=443
LOGGING_ES_PROMETHEUS_SERVICE_PORT_PROXY=443
LOGGING_ES_SERVICE_HOST=172.26.233.225
LOGGING_ES_SERVICE_PORT=9200
LOGGING_KIBANA_PORT=tcp://172.27.107.118:443
LOGGING_KIBANA_PORT_443_TCP=tcp://172.27.107.118:443
LOGGING_KIBANA_PORT_443_TCP_ADDR=172.27.107.118
LOGGING_KIBANA_PORT_443_TCP_PORT=443
LOGGING_KIBANA_PORT_443_TCP_PROTO=tcp
LOGGING_KIBANA_SERVICE_HOST=172.27.107.118
LOGGING_KIBANA_SERVICE_PORT=443



Version-Release number of selected component (if applicable): 3.11.0-0.10.0


How reproducible: Always


Steps to Reproduce:
1. Install logging 3.11.0-0.10.0
2. Attempt to login to kibana



Additional info:

Inventory:

[OSEv3:children]
masters
etcd


[masters]
ip-172-31-3-24

[etcd]
ip-172-31-3-24



[OSEv3:vars]
deployment_type=openshift-enterprise

openshift_deployment_type=openshift-enterprise
openshift_release=v3.11
openshift_docker_additional_registries=registry.reg-aws.openshift.com



openshift_logging_master_url=https://ec2-35-165-251-149.us-west-2.compute.amazonaws.com:8443
openshift_logging_master_public_url=https://ec2-35-165-251-149.us-west-2.compute.amazonaws.com:8443
openshift_logging_kibana_hostname=kibana.apps.0730-z7d.qe.rhcloud.com
oreg_url=registry.reg-aws.openshift.com:443/openshift3/ose-${component}:${version}
openshift_logging_image_version=v3.11
openshift_logging_install_logging=true
openshift_logging_es_cluster_size=1
openshift_logging_es_pvc_dynamic=true
openshift_logging_es_pvc_size=50Gi
openshift_logging_es_pvc_storage_class_name=gp2
openshift_logging_fluentd_read_from_head=false
openshift_logging_es_memory_limit=8Gi
openshift_logging_curator_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_logging_kibana_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_logging_es_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_logging_install_eventrouter=true
Comment 1 Jeff Cantrill 2018-08-01 13:39:07 UTC 
This PR fixes Kibana not being able to connect to Elasticsearch but in the course of investigation found issues with permissions and seeding which I will investigate and address in subsequent BZ/PR
https://github.com/openshift/origin-aggregated-logging/pull/1269
Comment 4 Anping Li 2018-08-06 09:21:55 UTC 
It is using the correct router name With openshift3/ose-logging-kibana5/images/v3.11.0-0.11.0.0.  

{"type":"log","@timestamp":"2018-08-06T08:50:36Z","tags":["warning","elasticsearch","admin"],"pid":226,"message":"Unable to revive connection: https://logging-es:9200/"}
{"type":"log","@timestamp":"2018-08-06T08:50:36Z","tags":["warning","elasticsearch","admin"],"pid":226,"message":"No living connections"}

Note that,  you will hit https://bugzilla.redhat.com/show_bug.cgi?id=1595956.
Comment 6 errata-xmlrpc 2018-10-11 07:22:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2652